Abstract

     In many real-world scenarios, the utility of a user is derived from a single execution of a policy. In this case, to apply multi-objective reinforcement learning, the expected utility of the returns must be optimised. Various scenarios exist where a user’s preferences over objectives (also known as the utility function) are unknown or difficult to specify. In such scenarios, a set of optimal policies must be learned. However, settings where the expected utility must be maximised have been largely overlooked by the multi-objective reinforcement learning community and, as a consequence, a set of optimal solutions has yet to be defined. In this work, we propose first-order stochastic dominance as a criterion to build solution sets to maximise expected utility. We also define a new dominance criterion, known as expected scalarised returns (ESR) dominance, that extends first-order stochastic dominance to allow a set of optimal policies to be learned in practice. Additionally, we define a new solution concept called the ESR set, which is a set of policies that are ESR dominant. Finally, we present a new multi-objective tabular distributional reinforcement learning (MOTDRL) algorithm to learn the ESR set in multi-objective multi-armed bandit settings.

Introduction

     When making decisions in the real world, decision makers must make trade-offs between multiple, often conflicting, objectives [44]. In many real-world settings, a policy is only executed once. For example, consider a municipality that receives the majority of its electricity from local solar farms. To deal with the intermittency of the solar farms, the municipality wants to build a new electricity generation facility. The municipality are considering two choices: building a natural gas facility or adding a lithium-ion battery storage facility to the solar farms. Moreover, the municipality want to minimise CO2 emissions while ensuring energy demand can continuously be met. Given a new energy generation facility will only be constructed once, a full distribution over each potential outcome for capacity to meet electricity demand and CO2 emissions must be considered to make an optimal decision. The current state-of-the-art multi-objective reinforcement learning (MORL) literature focuses almost exclusively on learning polices that are optimal over multiple executions. Given such problems are salient, to fully utilise MORL in the real world, we must develop algorithms to compute a policy, or set of policies, that are optimal given the single-execution nature of the problem.

Conclusion and future work

     MORL has been highlighted as one of several key challenges that need to be addressed in order for RL to be commonly deployed in real-world systems [12]. In order to apply RL to the real world, the MORL community must consider the ESR criterion. However, the ESR criterion has largely been ignored by the MORL community, with the exception of the works of Roijers et al. [33, 36], Hayes et al. [15, 16] and Vamplew et al. [43]. The works of Hayes et al. [15, 16] and Roijers et al. [33] present single-policy algorithms that are suitable to learn policies under the ESR criterion; however, prior to this work, a formal definition of the necessary requirements to compute policies under the ESR criterion had not previously been defined. In Sect. 3, we outline, through examples and definitions, the necessary requirements to optimise under the ESR criterion. The formal definitions outlined in Sect. 3 ensure that an optimal policy can be learned when the utility function of the user is known under the ESR criterion. However, in the real world, a user’s preferences over objectives (or utility function) may be unknown at the time of learning [36].

     Prior to this paper, a suitable solution set for the unknown utility function scenario under the ESR criterion had not been defined. This long-standing research gap has restricted the applicability of MORL in real-world scenarios under the ESR criterion. In Sects. 4 and 5, we define the necessary solution sets required for multi-policy algorithms to learn a set of optimal policies under the ESR criterion when the utility function of a user is unknown. In Sect. 6, we present a novel multi-policy algorithm, known as multi-objective tabular distributional reinforcement learning (MOTDRL), that can learn the ESR set in a MOMAB setting when the utility function of a user is unknown at the time of learning. In Sect. 7, we evaluate MOTDRL in two MOMAB settings and show that MOTDRL can learn the ESR set in MOMAB settings. This work aims to answer some of the existing research questions regarding the ESR criterion. Moreover, we aim to highlight the importance of the ESR criterion when applying MORL to real-world scenarios. In order to successfully apply MORL to the real world, we must implement new single-policy and multi-policy algorithms that can learn solutions for nonlinear utility functions in various scenarios.

Abstract

     This study integrated multiple objective decision-making approaches including data envelopment analysis (DEA), rough set theory and TOPSIS method for exploring the competitiveness of pharmaceutical multinational enterprises. Firstly, this study applied an advanced two-stage network DEA to measure the R&D efficiency and business performance of pharmaceutical multinational enterprises (PMNEs) listed in the Forbes Global 2000 and ranked these PMNEs by using rough set theory, DEA and TOPSIS method. In addition, differences in environmental, social, and governance (ESG) performance across three continents were investigated. Findings show that North America is significantly the best (worst) region in terms of business performance (R&D efficiency) while Europe is significantly the best (worst) region in terms of R&D efficiency (business performance). Alfresa Holdings Corporation, a PMNE from Japan, is the only PMNE that is efficient from both an R&D and business performance perspective. European PMNEs have significantly better environmental and social performance than other regions, however, they have the worst governance performance. Overall, this study provides insights to managers and investors into the application of various methods for accurately measuring performance and ranking multinational enterprises.

Introduction

     The pharmaceutical industry is one of the most important and competitive industries in the world, it not only has a critical role in the health maintenance of individuals but also in economic growth (Sharma & Modgil, 2019). Pharmaceutical firms have spent a substantial proportion of their time and investment in research and development (R&D) (Rao, 2020). In 2018, R&D expenditure in the pharmaceutical industry was 179 billion U.S. dollars globally (Mikulic, 2020). A correct grasp of the R&D efficiency of the industry is required to improve resource allocation and thereby prevent either excess or shortage of resource inputs (Liu & Lyu, 2020). According to Lu et al. (2019), evaluating multiple indicators simultaneously is useful for appropriate decision-making. An overall efficiency can be decomposed into two stages of efficiencies (Kao & Hwang, 2008), suggesting that we should also evaluate the sub-processes of a firm (Kao & Hwang, 2011). That is, managers should evaluate their firm performance not only from a multidimensional perspective but also from those of its internal processes. As mentioned in the research of Sharma and Modgil (2019), business performance additionally depends on cost reduction of manufacturing operations and improving the firm’s innovative ability. Recently, Hsieh et al. (2020) have applied two-stage data envelopment analysis (DEA) to investigate organizations’ innovation and business performance by investigating the network structures of decision-making units (DMUs).

Conclusion

     This study measures the firm performance of 41 PMNEs from a two-stage perspective, namely R&D efficiency and business performance, by integrating a two-stage network DEA and SOCP technique. Moreover, this study integrates the DEA approach and rough set theory to rank PMNEs. The findings of this research can serve as a reference for managers of PMNEs in their decision-making process. Besides, we examine differences in the ESG scores of the PMNEs among different geographic regions (America, Asia, and Europe). This acts as a guide for investors, managers, and other stakeholders when attempting to move their ESG initiatives further.

     R&D performances in the operation of the observed PMNEs are not attracting enough attention. Alfresa Holdings Corporation is the only efficient PMNE for both R&D efficiency and business performance. North America is significantly the best (worst) region in terms of business performance (R&D efficiency) while Europe is significantly the best (worst) region in terms of R&D efficiency (business performance). From the perspective of ESG performance, European PMNEs significantly have better environmental and social performance than other regions whereas they are the worst concerning governance performance.

Abstract

     The rapid increase in the world population along with the rising trend of global economic growth triggers serious problems related to energy consumption. In this regard, a comprehensive analysis of the current and potential energy situation in the light of demographic trends, main economic indicators, technical conditions, social and environmental concerns has become a significant issue for developing well-planned global and regional energy strategies. In this study, a multi-objective decision-making model is constructed for renewable energy planning to determine the most appropriate resource diversity for Turkey by focusing on five renewable energy sources, namely solar, wind, geothermal, hydroelectric, and biomass. There are four objective functions presented in the proposed multi-objective model including maximization of the technical score of regions, maximization of the job creation, maximization of the environmental score, and minimization of the cost. As a solution methodology, a two-phase fuzzy goal programming approach is performed. The proposed model is solved using LINGO software as a comprehensive solver for model optimization. The appropriateness of the proposed method is analyzed by comparing it with other multi-objective methods. Results reveal that the majority of the energy demand should be met via solar and hydroelectric energies for the satisfaction of the indicated objectives.

Introduction

     The ongoing growth in the world population and the global economic development of countries together with the effects of industrialization, urbanization, and technological developments, have caused a dramatic increase in the amount of resource consumption. In the meantime, the level of consumption, the most inevitable of which is the need for energy, maintains its rise in many ways despite the fact that the world’s fossil resources are finite and rapidly depleting. Since the world already has limited reserves in terms of fossil fuel resources, and these resources are not evenly distributed between countries, it is not possible to meet the energy demand with the existing fossil resources in the short term. Moreover, the use of fossil fuels causes damage to the environment and living conditions, especially human health, animal life, and nature. This situation is also considered as another threat. Hence, it seems inevitable that the energy problem will be one of the major challenges humanity must deal with in the short term since energy is considered one of the most important subjects on the agenda of policymakers. In these circumstances, decision-makers have started to find out alternative sources to meet the energy needs. Energy sources in the renewable form are considered fundamental primary sources of alternative energy that require strategic planning in terms of sustainable development. Therefore, reducing the use of fossil resources and replacing them with renewable energy sources in a controlled manner are both critical for energy sustainability and vital for energy planning in the long term.

Conclusion

     Real-world problems in planning activities require consideration of multiple conflicting objectives simultaneously. In recent years, renewable energy planning has become a very important issue, which poses itself as a multi-objective problem due to its impact on economic, social, and environmental concerns. As the number of options increases, it becomes more difficult for decision-makers to determine specific renewable energy technology among alternatives [103]. Therefore, the proposed model for the prioritization of renewable energy sources based on twenty-one electricity distribution zones in Turkey has been developed in a multi-objective structure.

     The advantage of the proposed multi-objective model is that it can be further modified by integrating particular considerations such as national renewable energy targets and local incentive support schemes. In this way, the model enables the evaluation of several objectives simultaneously and better accommodates various parameters together in order to contribute to the process of determining the most suitable renewable energy technologies in future energy planning. Thus, this study presents a framework that provides a useful roadmap for decision-makers in the development and implementation of sustainable energy plans by analyzing renewable energy resources from a regional perspective to utilize energy potential efficiently. In addition, another contribution made through this study is the use of efficient solution methods comparatively in multi-objective analysis, which allows decision-makers to decide on the most appropriate evaluation method in the planning process. Hence, the two-phase fuzzy goal programming method is utilized for the evaluation procedures in comparison with three other multi-objective methods including lexicographic optimization, goal programming with absolute deviation, and goal programming with percentage deviation.

Piecewise linear functions have been used to model and solve non-linear problems in science, social science, and technology fields since the 1950‟s. Multi-choice goal programming (MCGP) has been widely used to solve multiple objective decision-making (MODM) problems in which each goal mapping with multiple aspiration levels is allowed, expanding the original feasible region to obtain better solutions in the MODM problems. This paper integrates the efficient S-shaped penalty method, arbitrary piecewise linear utility functions, trapezoidal utility functions, and MCGP to solve a topology design problem in a remote patient monitoring system (RPMS) by providing universal senior citizen coverage in which quantitative (e.g., patient satisfaction-related cost) and qualitative (e.g., satisfaction) issues are considered simultaneously. In addition, some novel utility functions such as the force utility function, indicator utility function, and arbitrary utility function are proposed to improve the usefulness of MCGP in the field of management science. These proposed utility functions can be easily used to model qualitative issues in real-world problems. A topology design problem for an RPMS is demonstrated to justify the feasibility, usefulness, and compatibility of the proposed methods. Further, sensitivity analysis and managerial implications are provided using an RPMS in Taiwan.

Introduction

Given the aging society and falling birth rate, Taiwan has been classified as an “aged” society since 2018 and will turn into a “hyper-aged” society by 2025. The increasing healthcare costs for the growing population of elderly, chronic diseases, as well as the demand for new technologies and treatments in addition to the decrease in the number of healthcare professionals compared to the elderly increase make the traditional healthcare service model inadequate. Conventional healthcare model inadequacy can be explained by the following two aspects: (1) large hospitals (e.g., accredited hospitals and medical centers) are always overcrowded, and as a result, people often don‟t have access to outpatient clinics and must wait a long time for inpatient care; and (2) increasing healthcare costs also place a considerable burden on the government‟s finances. Nowadays, establishing an efficient remote patient monitoring system (RPMS) to take care of the greater number of elderly people as well as reduce healthcare costs is essential in Taiwan. Moreover, it is very important to build an RPMS with a more complete topology network under a limited budget to take care of senior citizens. To implement the RPMS for serving senior citizens, several goals and criteria should be considered simultaneously. These goals and criteria contain both quantitative (e.g., the RPMS setup cost and investment cost) and qualitative (e.g., satisfaction of senior citizens) issues, and the balance between them (e.g., balancing the setup cost and the satisfaction of senior citizens), complicates the problem. This is a typical multiple objective decision making (MODM) problem with conflicting goals.

The goal of green supply chain management (GSCM) is a thorough integration of logistical and financial information to increase the competitiveness of supply chain units’ products or services, resulting in sustainable enterprise development and improved environmental protection (Wan et al., 2017; Sarkar et al., 2017). The leading global enterprises have recognized the urgency of the need to take measures for environmental protection and have begun to change their own policies and practices to conform to these goals. Thus, the procurement of raw materials in compliance with environmental regulations has already become a basis for the evaluation of green partners (Lu et al., 2007). Many in the manufacturing industry have adjusted their production concepts and fostered the introduction of environmental awareness into their organizations. Some have also, established environmental standards for the management of scrap products and take into consideration the recyclability of the raw materials obtained from their suppliers (Chen et al., 2012). Manufacturing firms typically spend approximately 60% of their total income on purchasing raw materials and components. For high-tech manufacturers, this amount can be as high as 80% (Kokangul and Susuz, 2009; Lee and Drake, 2010) meaning that purchasing strategies are very critical in GSCM. Selecting suitable firms as suppliers of products or services requires the consideration of numerous complex factors and thus qualifies as a multicriteria decision-making (MCDM) problem (Kumar et al., 2017). Given the current background of rising environmental protection consciousness, enterprises must pay attention to saving energy, waste elimination, parts interchangeability, ease of disassembly, and recyclability. Thus, green supplier management and purchasing are becoming increasingly critical (Lin et al., 2011). Manufacturers typically purchase the same raw materials or components from several suppliers, especially when a single supplier is unable to satisfy the demand due to capacity and risk-sharing limitations. This requires a compromise solution based on the manufacturer’s enterprise sustainability policy and environmental purchasing objectives (Govindan and Sivakumar, 2016). A few studies have discussed the integration of green supplier selection and order allocation. Most of these studies have applied MCDM models. In general, the MCDM process is categorized as multiple attribute decision-making (MADM) or multiple objective decision-making (MODM) on the basis of the problem’s solution space (i.e., whether the solution space is continuous or discrete) (Liou and Tzeng, 2012).