1- Introduction

2- Related work

3- Optimization by continuation

4- Results and discussion

5- Conclusions and recommendations

References

Abstract

Artificial Neural Networks research field is among the areas of major activity in Artificial Intelligence. Training a neural network is an NP-hard optimization problem that presents several theoretical and computational limitations. In optimization, continuation refers to an homotopy transformation of the fitness function that is used to obtain simpler versions of such fitness function and improve convergence. In this paper we propose an approach for Artificial Neural Network training based on optimization by continuation and meta-heuristic algorithms. The goal is to reduce overall execution time of training without causing negative effects in accuracy. We use continuation together with Particle Swarm Optimization, Firefly Algorithm and Cuckoo Search for training neural networks on public benchmark datasets. The continuation variations of the studied meta-heuristic algorithms reduce execution time required to complete training in about 5–30% without statistically significant loss of accuracy when compared with standard variations of the meta-heuristics.

Introduction

In the latest years, we have seen the efforts in optimization field to solve very complex and challenging tasks. At the same time, the success in this area has led researchers and practitioners to address much larger instances and more difficult classes of problems, specially in meta-heuristic area [1,2]. Training a neural network is an NP-hard optimization problem [3] that usually involves thousands and millions of parameters. Additionally, the fitness function of Artificial Neural Networks (ANNs) is highly non-convex [4] and presents a poor correspondence between its local and global structure [5]. Also, the number of local minima increases exponentially respect to the number of parameters of the network. In fact, such local minima appears to be saddle points instead of true minima [6] reducing the convergence speed of optimization, specially for gradient based training algorithms. In general, second-order methods have not obtained better results than first-order methods in this area [5]. These facts become even more problematic as we start using modern deep learning architectures. ANNs are widely used in several applications of image recognition [7], signal processing [8], speech [9] and several other fields Nevertheless, neural networks are playing a major role in the field of pattern recognition and learning representations from data. The most popular algorithm for training is Stochastic Gradient Descent (SGD) which is a gradient based first order1 method that has been proven to converge into local minima in the parameter space. In general, first and second order2 optimization methods require the calculation of derivatives and Hessians for which Automatic Reverse Mode Differentiation (ARMD) have to be used [10]. The parallel implementation of such methods faces important challenges due to the inherent sequential nature of ARMD. Recently, several meta-heuristic algorithms for optimization have been actively used in ANN training. Meta-heuristic algorithms have features to escape local minima and increase the probability of global convergence. In the literature there are reports of natureinspired meta-heuristics such as Cuckoo Search [11,12], Firefly Algorithm [13,14], Wolf Search Optimization [15,16], Particle Swarm Optimization (PSO) [17,18] and others.

Integer programming models for community detection in relational networks have diverse applications in different fields. From making our lives easier by improving search engine optimization to saving our lives by aiding in threat detection and disaster management, researches in this niche have added value to human experience and knowledge. Besides the community structure, the influential nodes or members in a complex network are highly effective at diffusing information quickly to others in the community. Prior research dealing with the use of optimization models for clustering networks has independently focused on detecting communities. In this research, we propose a new integer linear programming model to detect community structure in real-life networks and also identify the most influential node within each community. We validate the proposed model by testing it on a well-established community network. Further, the performance of the proposed model are evaluated by comparing it with the existing best performing optimization model as well as three heuristic approaches for community detection. The experimental results indicate that in most cases the proposed integer programming model performs better than the existing optimization model with respect to modularity, Silhouette coefficient and computational time. Besides, our model yields superior Silhouette and competitive modularity values compared to the heuristic approaches in many cases.

Introduction

A large number of complex systems can be expressed through networks, which is a group of nodes associated through edges or links (Raghavan et al., 2007). Social networks such as Facebook and Twitter are such examples, where the users are represented by the nodes and the ties/friendship between them is indicated by edges. Similarly, in supply chain logistics, nodes represent the supplier and customer locations, while edges denote the paths connecting the nodes. Due to its theoretical importance and practical applications, research on social, technological, physical and biological networks has gained importance in different branches of sciences. Representing the complex systems through relational networks provides an insight into the intricacy and the dynamic behavior of such systems (Pan et al., 2014). A network is said to exhibit a community structure if its nodes can be grouped into different clusters such that the nodes within the clusters are densely connected compared to the nodes outside the cluster (Radicchi et al., 2004). Finding communities in relational networks takes us one step further to better understanding complex systems, which in turn results in numerous benefits and applications (Girvan and Newman, 2002). Communities are formed by clustering the nodes in a relational network, where the nodes (or objects) are placed into groups such that the objects in the same group have maximum similarities, while the ones in different groups have the least in common.

Most of the existing artificial neural networks (ANNs) applications, particularly for commercial environment, are developed as software. Yet, the parallelism offered by hardware may deliver some advantages such as higher speed, reduced cost, and higher tolerance of faults (graceful degradation) [1, 2]. Among various developed methods of ANNs implementations in field programmable gate arrays (FPGAs), e.g., [3 – 6], there is a breed of implementation which allows the structure of the ANN (i.e., the number of layers and/or neurons, etc.) to be altered without the need of re-synthesizing and re-implementation of the whole FPGA project. This feature increases the ANNs implementation flexibility to the similar level as offered by software, at the same time maintaining the advantages delivered by hardware. Unfortunately, existing solutions, e.g., [7 – 9], are based on fixed point arithmetic, have strongly limited calculations accuracy of the activation function, and require dedicated software tools for the formulation of a set of user instructions controlling the ANN calculations in the developed hardware. Some of them [9, 10] do not employ parallel architecture exploiting only a single neuron block for the calculations of the whole ANN. In the case of [10] floating point (FP) arithmetic is used and a relatively high accuracy of the activation function is achieved, however the feasibility of the alteration of the ANN structure without reimplementation of the whole project is heavily compromised. In this brief paper the FPGA implementations of feed forward ANNs, namely the resource-saving and parallel, are presented. The resource-saving implementation characterizes considerably lower calculations speed than the parallel one, but requires remarkably lower number of FPGA resources. Both implementations employ single precision floating point arithmetic and apply a very high accuracy algorithm for the activation function calculation with the Padé approximation of the exponential function. This enables the direct exploitations of the ANN’s weights and biases values calculated off-line, e.g., by the Matlab software. The important feature of the proposed implementations is that the structure of the ANN can easily be changed (even on-line during the system operation) by the replacement of the FPGA Block RAM memory content without the usage of FPGA synthesis tools. The aforementioned traits of the developed implementations make them a solid and versatile candidate for a hardware accelerator of ANNs calculations. Particularly, the Padé approximation of the exponential function as well as the usage of the block RAM memory for the ANN’s structure definition also constitute a novelty of the proposed solution.

This paper reports on how the technology behind cryptocurrency (Bitcoin) i.e. Blockchain could offer its services in distributed energy system (DES), noting on the issues related to operating conditions, energy generation monitoring, energy sharing and trading, financial flows, emission inventory, carbon emission trading and many more. Information on blockchain use cases and implications in DES is also reported.

Introduction

Most of us who have been associated with the energy sector, have aware of the energy usage in early years before the electricity has taken birth. We will be able to visualize the days with hot, smoky, and indistinct lighting provided by the gas lamps. With the technological advancement, the concept of electric bulb has driven the electric sector to a massive utility scale leading to the development of a centralized energy network (CEN). Since its creation, the structure of CEN has not seen much difference and largly remains unchanged executing its duties in managing the energy generation, transmission, and distribution. In later stages due to various factors, the demand for electricity has increased, allowing rapid extension in the generation, transmission, and distribution aspects without much alteration in the CEN. It is an observed source from the energy industry, that there exists many prevailing issues due to the complex structure of CEN. These issues include reliability, outdated grid infrastructure, stability, energy loss, obsolete design, environmental concerns, lower efficiency in generation, transmission and distribution.

Abstract

This work studies the effectiveness of query expansion for email search. Three state-of-the-art expansion methods are examined: 1) a global translation-based expansion model; 2) a personalized-based word embedding model; 3) the classical pseudo-relevance-feedback model. Experiments were conducted with two mail datasets extracted from a large query log of a Web mail service. Our results demonstrate the significant contribution of query expansion for measuring the similarity between the query and email messages. On the other hand, the contribution of expansion methods for a well trained learning-to-rank scoring function that exploits many relevance signals, was found to be modest.

Introduction

Searching over email data has attracted a lot of attention recently and several attempts have been made by the research community to apply up-to-date ranking paradigms for email search [2, 4]. In these paradigms, the relevance of the message to the query is estimated by a complicated scoring function that considers many signals of relevance, including the message freshness, the textual similarity to the query, the user interaction with the message, and many more signals [2]. However, email queries which are extremely short become a severe limitation for an accurate estimation of message relevance to the query. While the average query length on the Web is about three terms per query, the average length in the email domain is only 1.5 terms per query [2].

In the Big Data era users require new technology to process data stream with high speed and variety of data type. Besides the traditional stream processing technology, it is important to catch the relations inside online streaming data. Most of the data streams are composed of primitive events that produced by sensor networks, internet, social networks, etc. The semantic information inside primitive events is quite limited. In real application, people usually pay more attention to higher level information such as business logic and rules. For example, enormous events are generated in a trading system, but a fraud detecting system only care about the events that can cause frauds. Complex Event Processing (CEP) [1] is the technology that interprets and combines streams of primitive events to identify higher level composite events. CEP has been used in many areas, such as sensor networks for environmental monitoring, continuous analyzing of stocks to detect trends, etc. Predictive Complex Event Processing is the technology to identify events before they have happened, so that they can be eliminated or their effects mitigated [2]. For example, a financial institution wishes to detect frauds or a financial regulator wishes to catch illegal trading patterns in advance. Another example is to predict the traffic status in road networks and take some actions proactively to mitigate or eliminate undesired future states. Simple predictive CEP can be supported by rule-based method which means users define some patterns and the system then continuously monitor the event streams to predict future events. However, for complex cases it is not easy to define predictive CEP patterns exactly. In such circumstance, predictive analytics technology can be applied to support predictive CEP. Predictive analytics applies several statistical and data mining techniques such as clustering, classification, regression and so on. Recently, neural networks (especially deep neural networks) and Bayesian Networks (BN) are commonly used as prediction models. BN and it variations, such as Dynamic Bayesian Networks (DBN) [3], Adaptive Bayesian Networks (ABN) [4], etc., are widely used in predictive analytics because they have the following advantages: (i) they allow to express directly the fundamental qualitative relationship of direct causation, (ii) there exists mathematical methods to estimate the state of certain variables given the state of other variables, (iii) there are methods in order to explain to the user how the system came to its conclusions [5]. Currently predictive CEP with predictive analytics has some challenges. First, traditional predictive analytics methods are designed for database which means they assume all data is available at any time. However, in predictive CEP the system can only process data on single-pass and cannot control over the order of samples that arrive over time. Stream-based predictive analytics methods are needed which is more difficult. Second, the distribution of data can change over time. A model learned from historical data may not fit the new coming data well. This means real time modeling and learning algorithms are needed. In order to support predictive CEP with BN, an Evolving Bayesian Networks (EBN) model is needed that can adjust itself automatically when the distribution of data changes.

Sustainable development is beeing applied emergently by transport authorities and in other activity sectors and industries all over the World. It is proponted by iniciatives which include environmental variables and social responsibility in strategic management of companies (Doerr, 2011). This is port case. Port sustainability is rooted in the proposals of the GRI (Global Reporting Initiative, 2000) and it preserves the four main ideas or dimensions which knock into shape sustainable development – institutional, economic, environmental and social dimensions- It is considered that sustainable management of a company or organism has as main target equilibrate keeping of its function and activity for a long time. So, it is necessary looking for an equilibrate development of economic, social, environmental and institutional dimensions (Serrano, 2015). This concept is beeing applied emergently by transport authorities and in other activity sectors and industries all over the World. It is proponted by iniciatives which include environmental variables and social responsibility in strategic management of companies (Doerr, 2011). This is port case. Port sustainability is rooted in the proposals of the GRI (Global Reporting Initiative, 2000) and it preserves the four main ideas or dimensions which knock into shape sustainable development – institutional, economic, environmental and social dimensions- (Fig. 1). It is considered that sustainable management of a company or organism has as main target equilibrate keeping of its function and activity for a long time. So, it is necessary looking for an equilibrate development of economic, social, environmental and institutional dimensions (Serrano, 2015). In this context, maritime transport requires an especial attention, because it transports over 80% foreign trade (tonelates-kilometres) (S
anchez et al., 2015). So, sustainable management must be understood in port sector as “management which allows containers traffic, solid and liquid granels, general goods and number of passenger grow up at the same time that energy and natural resources purchase, rubbish volume and negative impacts over social systems and ecosystems decreases in port influence areas” (Crespo Soler et al., 2005). However, one of the principal challenges to integrate sustainable criterias in management model and port development is broken out present inertia which considers economic factor as unique development variable. It is necessary that environmental and social variables gain importance to make port development travels to sustainability (Grupo de Trabajo 23, 2004). Spanish State Ports and Merchant Navy Law includes sustainability as one of the principles which must regulate planning model and port management. Article 55.4 foresees company planning of each Port Authority must be gone with a sustainability report-it is considered as an analysis and diagnostic tool-. This report use a methodology which is based on GRI’s one, but includes specific quality indicators and must be approved by Spanish State Ports and Port Authorities. Redaction of these reports is an important effort to integrate information about Port Authority behavior and information about its environmental, economic and social performance. However, this report does not determine Port Authority behavior, it only describes results using performance quality indicators (Crespo Soler et al., 2007).

Recently research of Yan & Huang who are the winners of PASCAL 2010 classification competition indicates that features are the key to the progress in recognition[1, 6]. And actually the description of features is an abstraction of the object needs to learn. Generally speaking, this kind of abstraction can be viewed as an impression[4]. For example, in a recognition task, one is given a picture and asked whether the animal in it is a cat. Usually, people can judge by their impressions of cats. For instance, in someone’s impression, cats have whiskers and are furry with sharp claws. They can see in near darkness and can hear very faint sounds made by mice. In these impressions, some of them are easy to quantify and others are not. Since descriptions of features by computers are different from impressions of humans, computers cannot handle the abstract information if without a suitable model for those features hard to quantify[9]. Deep learning is a tool which can learn features automatically and uses a deep model with internal weights rather than a single value or vector to express those learned features[2, 8]. The structure of deep model can extract abstract features for subsequent classifiers[10]. Moreover, it is worth mentioning that descriptions of features are not same in different levels or different views, which is similar to the idea of granular computing[11]. As a multi-level example, a blind man feels an elephant, only touching some part of it, and concluding what the elephant is like. The man draws an impression on the basis of partial understanding, and the overall judgement is composed of each part of the impression. We obtain various impressions based on one-side viewpoint and each view supplies amount of information which is merged into a concept by a certain structure. In addition, during the understanding of objective things, there is an idea of multilevel processing[5]. For example, when children learn the concept of “cat”, they first notice various kinds of pictures and those pictures supply the pixel information from the lowest level, and then they will use this information to conclude several partial features of cats, such as arms and legs, head, whiskers and eyes. These features are established on the bottom pixels, and they are relatively more abstract and with a higher level. Up above, these features are continued to be combined to a much higher level of features, such as body shape and textures, until to the top decision level to determine whether the animal in the picture is a cat[3]. The idea of multilevel and multi-view is reflected in the deep learning algorithms. Thus, we can develop an abstract deep learning model with multilevel connection layers.