With the continuous development of Internet technology and electronic information technology, big data technology and cloud computing technology also rise and develop, and have a positive impact on people’s lives. Data mining system can deeply mine the value information contained in big data, so as to assist users to solve practical problems and help users to make correct decisions and judgments. This paper presents an energy analysis of data mining algorithm based on cloud platform for Internet of things (IoT). First of all, an improved Apriori algorithm is proposed, which is based on Boolean matrix and sorting index rules. Then Boolean matrix is obtained after scanning the dataset and the Boolean matrix is preprocessed to delete the useless transactions and the item set, which are combined with sorting index to produce other item sets, effectively improving the efficiency of frequent item mining, which effectively reduce the memory usage. Secondly, the density-based spatial clustering of applications with noise (DBSCAN) clustering algorithm needs human intervention in the global parameter selection, and the process of regional query is complex and the query is easy to lose objects. An improved parameter adaptive and regional query density clustering algorithm is proposed, which can effectively delete the redundant data in the high-level complex data space on the premise of retaining the internal nonlinear structure of the IoT data. The efficiency of clustering is also improved accordingly. Finally, the simulation based on cloud platform verifies the effectiveness and superiority of the algorithm.

Introduction

With the continuous development of Internet and the intelligent terminals, the amount of data has become more and more, and the types of data have become extremely rich, which represents that human beings have been big data age [1]. In recent years, data analysis of the Internet of things (IoT) has attracted more and more [2]. Through effective data analysis methods, the characteristics and rules of the Internet of things can be mined to reduce the loss in each life cycle of the Internet of things [3]. At present, the traditional Internet of things data analysis method needs to establish a high-precision mathematical model of the IoT data system, and the system is limited to the linear and time invariant system [4]. At the same time, it has high requirements for knowledge in the professional field and low precision shortcomings, so it has great limitations in application. At present, the collection mechanism of Internet of things data in many areas is becoming more and more perfect [5]. The Internet of things database will collect Internet of things data, which contains a lot of undiscovered massages, and the Internet of things data has many disadvantages, but it is difficult for traditional Internet of things data analysis methods to find and summarize the content of these Internet of things data [6]. Knowledge discovery, machine learning and data mining technology are just new disciplines born under this background. Because data mining technology has the ability to deal with large amount of data and find potential massages, it is a way to mine important massages in IoT data [7].

1. Introduction

2. Background

3. Proposed Solution

4. Conclusion

Acknowledgement

References

The rapid development of the Internet, especially mobile Internet, makes it much easier for people to make social contacts online. Nowadays they tend to spend more and more time on social network service, producing a lot of image files. This brings a challenge to traditional standalone framework on handing the continued increasing image files. Therefore, it is advisable to find a new way to face the challenge. Hadoop is a notable, widely-used project for distributed storage and computations with high efficiency, data integrity, reliability and fault tolerance. Hadoop Distributed File System and MapReduce are two primary subprojects respectively for big data storage and computations. However, Hadoop do not provide any interface for image processing. Worse, both Hadoop Distributed File System and MapReduce have trouble processing large amount of small files, decreasing efficiency of files access and distributed computations. This prevents us from performing images processing actions on Hadoop. This paper proposes a method to optimize small image files storage on Hadoop and self-defines an input/output format to enable Hadoop to process image files.

Introduction

At the meantime, the rapid development of mobile technology and the wide spread of the Internet give rise to the time people spend on social networking service every day. They chat and share their life by pictures or videos with friends whenever they access to the net. Take Weibo as an example. According to The Report of Weibo Users Development 2016[2], the number of monthly active users on Weibo have reached 297 million while daily active users have got to 132 million by September, 2016. The report also revealed that these users active on Weibo mainly write tweet in the form of images with some text description, which accounts for 60%. We can simply estimate that even if we take Weibo alone into consideration, about 70 million of images will be created, stored and uploaded every day. And clearly there are many other network services similar to Weibo, popular and producing large amount of data especially images data. Thus, it is hard for traditional standalone framework to store and handle such a large number of image files. A solution to this problem is needed.

1- Introduction

2- Related work

3- Hadoop structure

4- Investigated algorithms

5- Experimental environment and results

6- Conclusions

References

Abstract

The advent of Solid State Drives (SSDs) stimulated a lot of research to investigate and exploit to the extent possible the potentials of the new drive. The focus of this work is on the investigation of the relative performance and benefits of SSDs versus hard disk drives (HDDs) when they are used as underlying storage for Hadoop’s MapReduce. In particular, we depart from all earlier relevant works in that we do not use their workloads, but examine MapReduce tasks and data suitable for performing analysis of complex networks which present different execution patterns. Despite the plethora of algorithms and implementations for complex network analysis, we carefully selected our “benchmarking methods” so that they include methods that perform both local and network-wide operations in a complex network, and also they are generic enough in the sense that they can be used as primitives for more sophisticated network processing applications. We evaluated the performance of SSDs and HDDs by executing these algorithms on real social network data and excluding the effects of network bandwidth which can severely bias the results. The obtained results confirmed in part earlier studies which showed that SSDs are beneficial to Hadoop. However, we also provided solid evidence that the processing pattern of the running application has a significant role, and thus future studies must not blindly add SSDs to Hadoop, but they should build components for assessing the type of processing pattern of the application and then direct the data to the appropriate storage medium.

Introduction

A complex network is a graph with topological features such as scale-free properties, existence of communities, hubs, and so on that is used to model real systems, for example, technological (Web, Internet, power grid, online social networks) networks, biological networks (gene, protein), social networks [23]. The analysis of online social networks (OSNs) such as Facebook, Twitter, Instagram has received significant attention because all these networks store and process colossal volumes of data, mainly in the form of pair-wise interactions, thus giving birth to networks, i.e., graphs which record persons’ interactions whose analysis and mining offers both operational and business advantages to the OSN owner. Modern OSNs are comprised by millions of nodes and even billions of edges; therefore any algorithm for their analysis that relies on a single machine (centralized) – exploiting solely the machine’s main memory and/or its disk – is eventually doomed to fail due to lack of resources. Thus, the digitization of the aforementioned relationships produces a vast amount of collected data, i.e., big data [9] requiring extreme processing power that only distributed computing can offer. However, developing a distributed solution is a challenging task because it must deal sometimes with sequential processes. Some analysis algorithms based on distributed solutions that can run only on a small cluster of machines are still insufficient, since modern OSNs are maintained by Internet giants such as Google, LinkedIn and Facebook who own huge datacenters and operate clusters of several thousand machines. These clusters are usually programmed by data-parallel frameworks of the MapReduce type [4], a big data analytics platform. The Hadoop [29] middleware was designed to solve problems where the “same, repeated processing” had to be applied to peta-scale volumes of data. Hadoop’s initial design was based on magnetic disk’s characteristics, enforcing sequential read and write operations introducing its own distributed file system (HDFS – Hadoop Distributed File System) with blocks of large size. Recently with the advent of faster Solid State Drives (SSDs) research is emerging to test and possibly to exploit the potential of the new technologically advanced drive [11, 12, 21, 33]. The lack of seek overhead gives them a significant advantage with respect to Hard Disk Drives (HDDs) for workloads whose processing requires random access instead of sequential access.

Nearly two decades after its emergence, the Cloud Computing remains gaining traction among organizations and individual users. Many security issues arise with the transition to this computing paradigm including intrusions detection. Intrusion and attack tools have become more sophisticated defeating traditional Intrusion Detection Systems (IDS) by large amount of network traffic data and dynamic behaviors. The existing Cloud IDSs suffer form low detection accuracy, high false positive rate and high running time. In this paper we present a distributed Machine Learning based intrusion detection system for Cloud environments. The proposed system is designed to be inserted in the Cloud side by side with the edge network components of the Cloud provider. This allows to intercept incoming network traffic to the edge network routers of the physical layer. A time-based sliding window algorithm is used to preprocess the captured network traffic on each Cloud router and pass it to an anomaly detection module using Naive Bayes classifier. A set of commodity server nodes based on Hadoop and MapReduce are available for each anomaly detection module to use when the network congestion increases. For each time window, the anomaly network traffic data on each router side are synchronized to a central storage server. Next, an ensemble learning classifiers based on the Random Forest is used to perform a final multi-class classification step in order to detect the type of each attack. Various experiment are performed in the Google Cloud Platform in order to assess the proposed system using the CIDDS-001 public dataset. The obtained results are satisfactory when compared to a standard Random Forest classifier. The system achieved an average accuracy of 97%, an average false positive rate of 0.21% and an average running time of 6.23s.

Introduction

The appealing features of Cloud computing continue to fuel its integration in many sectors including industry, governments, education, entertainment, to name few [1]. Cloud computing aims to provide convenient, on-demand, network access to a shared pool of configurable computing resources, which can be rapidly provisioned and released with minimal management effort or service provider interactions [2]. The pay-as-you-go and the on-demand elastic operation Cloud characteristics are changing the enterprise computing model, shifting on-premises infrastructures to off premises data centers, accessed over the Internet and managed by cloud hosting providers. However, many security issues arise with the transition to this computing paradigm including intrusions detection. Regardless the important evolution of the information security technologies in recent years, intrusions and attacks continue to defeat existing intrusion detection systems in Cloud environments [3]. Attackers developed new sophisticated techniques able to brought down an entire Cloud platform or even many within minutes. New records are breached each year by attacker. Recently a destructive DDoS attack have brought down more than 70 vital services of Internet including Github, Twitter, Amazon, Paypal, etc. Attackers have taken advantages of Cloud Computing and Internet of Things technologies to generate a huge amount of attack traffic; more than 665 Gb/s [4, 5]. Intrusion and attack tools have become more sophisticated challenging existing Cloud IDSs by large volumes of network traffic data, dynamic and complex behaviors and new types of attacks. It is clear that an IDS for Cloud should analyze large volumes of network traffic data, detect efficiently the new attack behaviors and reach high accuracy with low false. However preprocessing, analyzing and detecting intrusions in Cloud environments using traditional techniques have become very costly in terms of computation, time and budget. Therefore, efficient intrusions detection in Cloud environments requires adoption of new distributed and intelligent techniques such as Machine Learning techniques.

1- Introduction

2- Related work

3- Proposed framework

4- Proposed algorithm for seasonal climate change detection

5- Comparison of various change detection algorithms

6- Result and discussion

7- Conclusion

References

Abstract

Big data plays a vital role in the prediction of diseases that occur due to climate change. For such predictions, scalable data storage platforms and efficient change detection algorithms are required to monitor the climate change. However, traditional data storage techniques and algorithms are not applicable to process the huge amount of climate data. This paper presents a scalable data processing framework with a novel change detection algorithm. The large volume of climate data is stored on Hadoop Distributed File System (HDFS) and MapReduce algorithm is applied to calculate the seasonal average of climate parameters. Spatial autocorrelation based climate change detection algorithm is proposed in this paper to monitor the changes in the seasonal climate. The proposed climate change detection algorithm is compared with various existing approaches such as pruned exact linear time method, binary segmentation method, and segment neighborhood method.

Introduction

“Big Data” is defined by volume, velocity, and variety of data. Big data is very complex to process by traditional data processing techniques and tools. Nowadays, data generation sources like telescopes, satellite, sensor networks, social networks, wearable devices, mobile devices, streaming machines and high throughput instruments are continuously generating a large volume of data. Recently, big data analytics has been applied in various domains, such as healthcare, business process, scientific research, natural resource management, share marketing, social networking, community administration and climate modeling. Climate data is observed from various advanced sensor technologies and is used to represent the seasonal changes. Weather data collected from different climate laboratory and advanced computing technologies are used to give valuable information to the world. Meteorological data is most often used to predict the weather and other climate-related phenomena. In addition, climate data is also used for various purposes that lead to a significant development in weather forecasting, rocket launching, and public health. However, climate data collected from various sources are used to identify the seasonal changes. In general, the climate laboratories generate data in unstructured format. Statistical techniques or machine learning algorithms are used to get meaningful information from the raw data. For example, statistical techniques are used to identify the number of precipitation days for a specific region. In this regard, the World Climate Data Monitoring Programme (WCDMP) is developed by WMO’s World Climate Programme (WCP) that focuses on management and collection of large climate data observed from the global climate system [1].

In recent decades, increasingly large amounts of data are generated from a variety of sources. The size of generated data per day on the Internet has already exceeded two exabytes [1]. Within one minute, 72 h of videos are uploaded to Youtube, around 30.000 new posts are created on the Tumblr blog platform, more than 100.000 Tweets are shared on Twitter and more than 200.000 pictures are posted on Facebook [1]. Big Data problems lead to several research questions such as (1) how to design scalable environments, (2) how to provide fault tolerance and (3) how to design efficient solutions. Most existing tools for storage, processing and analysis of data are inadequate for massive volumes of heterogeneous data. Consequently, there is an urgent need for more advanced and adequate Big Data solutions. Many definitions of Big Data have been proposed throughout the literature. Most of them agreed that Big Data problems share four main characteristics, referred to as the four V’s (Volume, Variety, Veracity and Velocity) [2]. The volume refers to the size of available datasets which typically require distributed storage and processing. The variety refers to the fact that Big Data is composed of several different types of data such as text, sound, image and video. The veracity refers to the biases, noise and abnormality in data. The velocity deals with the place at which data flows in from various sources like social networks, mobile devices and Internet of Things (IoT). In this paper, we first give an overview of most popular and widely used Big Data frameworks which are designed to cope with the above mentioned Big Data problems. We identify some key features which characterize Big Data frameworks. These key features include the programming model and the capability to allow for iterative processing of (streaming) data. We also give a categorization of existing frameworks according to the presented key features. Then, we present an experimental study on Big Data processing systems with several representative batch, stream and iterative workloads. Extensive surveys have been conducted to discuss Big Data Frameworks [3,4,5]. However, our experimental survey differs from existing ones by the fact that it considers performance evaluation of popular Big Data frameworks from different aspects. In our work, we compare the studied frameworks in the case of both batch processing and stream processing which is not studied in existing surveys. We also mention that our experimental study is concluded by some best practices related to the usage of the studied frameworks in several application domains. More specifically, the contributions of this paper are the following: • We present an overview of most popular Big Data frameworks and we categorize them according to some features. • We experimentally evaluate the performance of the presented frameworks and we present a comparative study of them in the case of both batch processing, stream processing. • We highlight best practices related to the use of popular Big Data frameworks in several application domains. The remainder of the paper is organized as follows. In Section 2, we present existing surveys on Big Data frameworks and we highlight the motivation of our work. In Section 3, we discuss existing Big Data frameworks and provide a categorization of them. In Section 4, we present a comparative study of the presented Big Data frameworks and we discuss the obtained results. In Section 5, we present some best practices of the studied frameworks. Some concluding points are given in Section 6.