In recent years, learning management systems (LMSs) have played a fundamental role in higher education teaching models. A new line of research has been opened relating to the analysis of student behavior within an LMS, in the search for patterns that improve the learning process. Current e-learning platforms allow for recording student activity, thereby enabling the exploration of events generated in the use of LMS tools. This paper presents a case study conducted at the Catholic University of Murcia, where student behavior in the past four academic years was analyzed according to learning modality (that is, oncampus, online, and blended), considering the number of accesses to the LMS, tools employed by students and their associated events. Given the difficulty of managing the large volume of data generated by users in the LMS (up to 70 GB in this study), statistical and association rule techniques were performed using a Big Data framework, thus speeding up the statistical analysis of the data. The obtained results are demonstrated using visual analytic techniques, and evaluated in order to detect trends and deficiencies in the use of the LMS by students.

Introduction

A current tendency in higher education consists of the analysis and processing of data relating to the activity generated by users through the use of learning management systems (LMSs). The significant amount of data extracted from 5 these platforms provide fundamental information that can aid both teachers and students in improving their educational goals. One of the main problems at present is the analysis of this information, owing to two main factors: the already mentioned large volume of data available, and the different formats of these data, particularly for the management of unstructured data. 10 According to several studies (see, for example, [1, 2]), there exists a need for analytical tools to help to interpret LMS data and provide new knowledge for improving and even designing new e-learning techniques and methodologies. Before manipulating such information, it is also important to explore and select the necessary data from the LMS, according to the goals to be achieved. 15 The main objective of this work is to design and implement a framework based on big data technologies to identify the behavior patterns of LMS users and illustrate them in an intuitive and intelligible manner. For this purpose, we define the following steps: • Data preprocessing, by studying the data to be extracted from the LMS 20 and its storage in a big data platform. • Data analysis and identification of pattern recognition techniques that may provide value in the educational context. • Presentation of the obtained results according to suitable visual analytics techniques and tools. 25 For developing these steps, we have considered data processing guided by e-learning analytics, in which the connections among educational techniques, learning concepts and educational data mining are studied [3, 4]. Within this field, the areas most relevant to our work are learning analytics and visual analytics. The former aids us in data processing for discovering connections among 30 students, teachers and the learning process, with the purpose of creating recommendations that improve the overall educational process. The latter uses visual interfaces to illustrate the results obtained from analytical reasoning, facilitating an understanding of the new knowledge and aiding the users in discovering new relations or possible irregularities [5]. Here, we take a further step forward 35 in the use of e-learning analytics by integrating big data techniques into the educational data analysis.

Owing to its scalability and high fault-tolerance even on a distributed environment built up with personal computers, MapReduce has been introduced to parallelise the training of Back Propagation Neural Networks (BPNNs) on high-dimensional big datasets. Based on the evolution of local BPNNs produced by distributed Map tasks with different data splits, the paper proposes a novel approach to the distributed data-parallel training of BPNNs in MapReduce. The approach provides a reasonable measure to get global convergent BPNN candidates from local BPNNs only convergent on the specific data splits. Further, it not only can reduce the iterations to get the global convergent BPNN, but also shows great advantages in avoiding the training to get trapped into a local optimum on high-dimensional big datasets. To improve the training efficiency further, local BPNNs from the same computing node are merged based on the average of their weight matrices before they act as individuals of the population for the global evolution. Our approach also leverages Random Project based sampling techniques to evaluate the fitness of each individual in order to lower the computation cost in the evolution stage. Experiments show that our proposed approach improves the training efficiency highly compared to the stand-alone or traditional MapReduce BPNN training, and improves model accuracy for larger datasets. The comparison with 23 other popular classification approaches also shows that our proposed approach has big advantages in accuracy.

Introduction

An Artificial Neural Network (ANN) is a computer model to essentially mimic the knowledge acquisition and organisational skills of the human brain, which consists of a number of interconnected processing elements called neurons [1]. The neurons of an ANN are usually arranged into two or more layers logically, and interact with each other via weighted connections. These scalar weights determine the nature and strength of the influence between the interconnected neurons. Each neuron can be connected to all the neurons in the next layer. There is an input layer where data are presented to the neural network, and an output layer that holds the response of the network to the input. It is the intermediate layers, also known as hidden layers, that enable these networks to represent and compute complicated associations between patterns. Neural networks essentially learn through the adaptation of their connection weights according to input data [2]. Back Propagation Neural Network (BPNN), one of the most popular ANNs, employs the back-propagation algorithm for its connection weight adaptation and can approximate any continuous nonlinear functions by arbitrary precision with enough number of neurons [3]. We call this process the training of a neural network and the input data containing potential patterns is called training samples. In the past decades, ANNs have been widely used to model uncertain nonlinear functions [4], [5], and have shown great advantages in pattern recognition, classification and modelling of nonlinear relationships involving a multitude of variables [6].

1- Introduction

2- Related work

3- Hadoop structure

4- Investigated algorithms

5- Experimental environment and results

6- Conclusions

References

Abstract

The advent of Solid State Drives (SSDs) stimulated a lot of research to investigate and exploit to the extent possible the potentials of the new drive. The focus of this work is on the investigation of the relative performance and benefits of SSDs versus hard disk drives (HDDs) when they are used as underlying storage for Hadoop’s MapReduce. In particular, we depart from all earlier relevant works in that we do not use their workloads, but examine MapReduce tasks and data suitable for performing analysis of complex networks which present different execution patterns. Despite the plethora of algorithms and implementations for complex network analysis, we carefully selected our “benchmarking methods” so that they include methods that perform both local and network-wide operations in a complex network, and also they are generic enough in the sense that they can be used as primitives for more sophisticated network processing applications. We evaluated the performance of SSDs and HDDs by executing these algorithms on real social network data and excluding the effects of network bandwidth which can severely bias the results. The obtained results confirmed in part earlier studies which showed that SSDs are beneficial to Hadoop. However, we also provided solid evidence that the processing pattern of the running application has a significant role, and thus future studies must not blindly add SSDs to Hadoop, but they should build components for assessing the type of processing pattern of the application and then direct the data to the appropriate storage medium.

Introduction

A complex network is a graph with topological features such as scale-free properties, existence of communities, hubs, and so on that is used to model real systems, for example, technological (Web, Internet, power grid, online social networks) networks, biological networks (gene, protein), social networks [23]. The analysis of online social networks (OSNs) such as Facebook, Twitter, Instagram has received significant attention because all these networks store and process colossal volumes of data, mainly in the form of pair-wise interactions, thus giving birth to networks, i.e., graphs which record persons’ interactions whose analysis and mining offers both operational and business advantages to the OSN owner. Modern OSNs are comprised by millions of nodes and even billions of edges; therefore any algorithm for their analysis that relies on a single machine (centralized) – exploiting solely the machine’s main memory and/or its disk – is eventually doomed to fail due to lack of resources. Thus, the digitization of the aforementioned relationships produces a vast amount of collected data, i.e., big data [9] requiring extreme processing power that only distributed computing can offer. However, developing a distributed solution is a challenging task because it must deal sometimes with sequential processes. Some analysis algorithms based on distributed solutions that can run only on a small cluster of machines are still insufficient, since modern OSNs are maintained by Internet giants such as Google, LinkedIn and Facebook who own huge datacenters and operate clusters of several thousand machines. These clusters are usually programmed by data-parallel frameworks of the MapReduce type [4], a big data analytics platform. The Hadoop [29] middleware was designed to solve problems where the “same, repeated processing” had to be applied to peta-scale volumes of data. Hadoop’s initial design was based on magnetic disk’s characteristics, enforcing sequential read and write operations introducing its own distributed file system (HDFS – Hadoop Distributed File System) with blocks of large size. Recently with the advent of faster Solid State Drives (SSDs) research is emerging to test and possibly to exploit the potential of the new technologically advanced drive [11, 12, 21, 33]. The lack of seek overhead gives them a significant advantage with respect to Hard Disk Drives (HDDs) for workloads whose processing requires random access instead of sequential access.

Nearly two decades after its emergence, the Cloud Computing remains gaining traction among organizations and individual users. Many security issues arise with the transition to this computing paradigm including intrusions detection. Intrusion and attack tools have become more sophisticated defeating traditional Intrusion Detection Systems (IDS) by large amount of network traffic data and dynamic behaviors. The existing Cloud IDSs suffer form low detection accuracy, high false positive rate and high running time. In this paper we present a distributed Machine Learning based intrusion detection system for Cloud environments. The proposed system is designed to be inserted in the Cloud side by side with the edge network components of the Cloud provider. This allows to intercept incoming network traffic to the edge network routers of the physical layer. A time-based sliding window algorithm is used to preprocess the captured network traffic on each Cloud router and pass it to an anomaly detection module using Naive Bayes classifier. A set of commodity server nodes based on Hadoop and MapReduce are available for each anomaly detection module to use when the network congestion increases. For each time window, the anomaly network traffic data on each router side are synchronized to a central storage server. Next, an ensemble learning classifiers based on the Random Forest is used to perform a final multi-class classification step in order to detect the type of each attack. Various experiment are performed in the Google Cloud Platform in order to assess the proposed system using the CIDDS-001 public dataset. The obtained results are satisfactory when compared to a standard Random Forest classifier. The system achieved an average accuracy of 97%, an average false positive rate of 0.21% and an average running time of 6.23s.

Introduction

The appealing features of Cloud computing continue to fuel its integration in many sectors including industry, governments, education, entertainment, to name few [1]. Cloud computing aims to provide convenient, on-demand, network access to a shared pool of configurable computing resources, which can be rapidly provisioned and released with minimal management effort or service provider interactions [2]. The pay-as-you-go and the on-demand elastic operation Cloud characteristics are changing the enterprise computing model, shifting on-premises infrastructures to off premises data centers, accessed over the Internet and managed by cloud hosting providers. However, many security issues arise with the transition to this computing paradigm including intrusions detection. Regardless the important evolution of the information security technologies in recent years, intrusions and attacks continue to defeat existing intrusion detection systems in Cloud environments [3]. Attackers developed new sophisticated techniques able to brought down an entire Cloud platform or even many within minutes. New records are breached each year by attacker. Recently a destructive DDoS attack have brought down more than 70 vital services of Internet including Github, Twitter, Amazon, Paypal, etc. Attackers have taken advantages of Cloud Computing and Internet of Things technologies to generate a huge amount of attack traffic; more than 665 Gb/s [4, 5]. Intrusion and attack tools have become more sophisticated challenging existing Cloud IDSs by large volumes of network traffic data, dynamic and complex behaviors and new types of attacks. It is clear that an IDS for Cloud should analyze large volumes of network traffic data, detect efficiently the new attack behaviors and reach high accuracy with low false. However preprocessing, analyzing and detecting intrusions in Cloud environments using traditional techniques have become very costly in terms of computation, time and budget. Therefore, efficient intrusions detection in Cloud environments requires adoption of new distributed and intelligent techniques such as Machine Learning techniques.

The purpose of this research work is to explore various limitations of conventional search and page ranking systems in an E-Commerce environment. The key objective is to assist customers in making an online purchase decision by providing personalized page ranking order of E-Commerce web links in response to E-Commerce query by analyzing the customer preferences and browsing behavior. This research work first employs an orderly and category wise literature review. The findings reveal that conventional search systems have not evolved to support big data analysis as required by modern E-Commerce environment. This work aims to develop and implement second-generation HDFS- MapReduce based innovative page ranking algorithm, i.e. Relevancy Vector (RV) algorithm. This research equips the customer with a robust metasearch tool, i.e. IMSS-AE to easily understand personalized search requirements and purchase preferences of customer. The proposed approach can well satisfy all critical parameters such as scalability, partial failure support, extensibility as expected from next-generation big data processing systems. An extensive and comprehensive experimental evaluation shows the efficiency and effectiveness of proposed RV page ranking algorithm and IMSS-AE tool over and above other popular search engines.

Introduction

In this modern era of big data, the shopping activity is modified a lot because of enormous growth in online shopping websites, also known as E-Tailers. The new age customers prefer to shop through these online portals because of various attractions in countries like India such as easy and cheap availability of the Internet. The primary reason is intense competition between telecoms, for instance, Reliance Jio prime membership offers free unlimited internet data usage for three months for all of its users in nominal charges. Some of the other reasons include lucrative cash back and easy returns without deduction of shipping charges from portals like PayTm, Cash on delivery type regular features from E-Commerce sites like Flipkart, Amazon, and other E-Tailers. Moreover, searching a suitable E-Commerce website to best suit the customer purchase requirements is not so easy as customers are primarily dependent on conventional search engines like Google, Bing to find a suitable E-Commerce web site. However, when different users search the same E-Commerce query, even a most advanced and popular search engine retrieves the same result as discussed by Gomez-Nieto et al. (2014). Thus, irrespective of the background and personalized tastes of customer submitting the query as most of the modern search engines tend to return the results by interpreting the E-Commerce query in various possible ways. Moreover, if the query is ambiguous or incomplete, then the situation will get even worse as discussed by Malhotra and Verma (2013). For instance, for the incomplete E-Commerce search query ‘‘Galaxy”, some customers may be interested in links to buy a new Samsung Galaxy series mobile phone, while another customer may be interested in searching links for online booking of tickets for a movie Guardians of the Galaxy Vol. 2. Hence, there is an urgent need for personalized E-Commerce search system. The personalized system may modify the E-Commerce search query by keeping track of customer’s preferences by maintaining his/her profile, search preferences through browsing history, etc. over a period and return results in correct order of ranking with customer’s relevant output links on top to best suit the customer requirements (See Fig. 1).

1- Introduction

2- PRELIMINARIES

3- The Proposed Approach

4- Experimental setup and results

5- Conclusions

6- Acknowledgement

References

Abstract

In this paper, a big data analytic framework is introduced for processing high-frequency data stream. This framework architecture is developed by combining an advanced evolving learning algorithm namely Parsimonious Network Fuzzy Inference System (PANFIS) with MapReduce parallel computation, where PANFIS has the capability of processing data stream in large volume. Big datasets are learnt chunk by chunk by processors in MapReduce environment and the results are fused by rule merging method, that reduces the complexity of the rules. The performance measurement has been conducted, and the results are showing that the MapReduce framework along with PANFIS evolving system helps to reduce the processing time around 22 percent in average in comparison with the PANFIS algorithm without reducing performance in accuracy.

Introduction

The rapid growth of data generated through the Internet, which causes big data, attracts great attention from many stakeholders. This phenomenon takes place in many areas in the real life such as business, management, medical, government, and society administration. Big data is unique of its 4Vs characteristics: volume, velocity, variety, and veracity. Volume related to the number of data generated in the storage, which is associated with the scale of data. Velocity indicates the flow rate of continuous data which is associated with the data streams. Variety of data is char-acterized by the number of different format of the data, whereas veracity shows the uncertainty of data where the data sources need to be validated (trustees, accuracy, and data quality). Big data provides enormous opportunities for government/organizations in discovering and extracting valuable information/knowledge of their system which can be beneficial for them in the decision-making processes. In the business area for example, Wal-Mart collaborated with Hewlett Packard trace every purchase record belonging to their customers from their point-of-sales terminals, where their transactions reach around 267 million per day. This valuable transaction data becomes a key basis for company to improve their benefits by applying pricing strategy and advertising campaigns [10, 6]. In this case, the decision could be made by applying some techniques such as data mining, which is extensively used for many decision-making problems in many real-life applications. However, discovering meaningful insight of big data is challenging due to its 4V’s characteristics, which lie on the difficulties in data capture, data storage, data analysis and data visualization [37, 6]. Therefore, an advanced data mining techniques and technologies are highly necessary to process and analyze big data. Big data is often stored in cloud to support the extensibility and scalability of local storage refers to one characteristic of big data, namely volume. In order to extract valuable information of big data efficiently, there is an urgent demand to modify existing data mining techniques to be scalable in processing large-scale dataset. This issue led to a necessity to the development distributed or parallelize scenario in processing big data. In addition, big data are also generated by the arrival of new instances continuously in either by batches or one by one, known as data stream, emerging from real-world applications[9, 33]. Therefore, it is necessary for machine learning algorithm to adapt to rapidly changing non-stationary data streams. Note that stream processing/mining in the web news domain has been conducted in [39] using eT2Class [31], which is able to handle streaming data[4]. This phenomenon triggers the development of evolving learning algorithms, which are able to learn big data continuously [13] by evolving its model to adjust the shift and drift of the big data pattern.

Abstract

In the real world practice, software systems are often built without developing any explicit upfront model. This can cause serious problems that may hinder the almost inevitable future evolution, since at best the only documentation about the software is in the form of source code comments. To address this problem, research has been focusing on automatic inference of models by applying machine learning algorithms to execution logs. However, the logs generated by a real software system may be very large and the inference algorithm can exceed the processing capacity of a single computer.

Introduction

Software behavior models play an important role in the whole life cycle of software systems. Through models, software engineers may gain a deep understanding of how a system behaves without dealing with the intricacies of the implementation. Although good software engineering practices suggest that models should be developed upfront, before deriving an implementation, reality shows that often models do not exist, or they are inconsistent with the implementation. In fact, building a proper model is costly, hard, and requires both mathematical skills and ingenuity. Moreover, even if models are developed, they are often not updated with the changes in the implementation and therefore the models and the implementation progressively diverge. Model inference is a promising approach to tackle this problem by using machine learning to infer software behavior models automatically from execution logs [1–3]. Many model inference algorithms [4–6] have been proposed by recent research. To infer accurate models, the logs should contain as much detail information as possible. However, a log with more information also increases the difficulty of model inference task. The logs generated by real systems are usually very large. For example, Prospex [7] infers state machines from network logs for vulnerability analysis of network applications. In practice, network logs collected passively can be enormous, while models need to be inferred quickly to ensure timeliness of subsequent analyses.

Abstract

Spam, under a variety of shapes and forms, continues to inflict increased damage. Varying approaches including Support Vector Machine (SVM) techniques have been proposed for spam filter training and classification. However, SVM training is a computationally intensive process. This paper presents a MapReduce based parallel SVM algorithm for scalable spam filter training. By distributing, processing and optimizing the subsets of the training data across multiple participating computer nodes, the parallel SVM reduces the training time significantly. Ontology semantics are employed to minimize the impact of accuracy degradation when distributing the training data among a number of SVM classifiers. Experimental results show that ontology based augmentation improves the accuracy level of the parallel SVM beyond the original sequential counterpart.

Introduction

The overall email ecosystem has become the most ubiquitous modern day communication tool. Its popularity as well as widespread availability have also created an opportunity for a lucrative business model based on unsolicited bulk email, or rather spam. The proliferation of spam has reached widespread proportions. Spam damages enabling communication infrastructures as well as lumbers consumers and service providers in its trail. Such impact and subsequently the importance to identify better ways to control and mitigate its consequences are reflected in the attention that spam continuously gets from various perspectives. Machine learning techniques such as Support Vector Machines (SVMs) have been applied in spam filtering [1–5]. SVM training is a computationally intensive process primarily due to its convex quadratic programming challenges associated with the dense Hessian Matrix involved during optimization. Numerous SVM formulations, solvers and architectures for improving SVM performance have been explored and proposed [6,7] including the Message Passing Interface (MPI) based parallel approaches [8–10]. SVM decomposition is another technique for improving SVM training performance [11–13]. Less conventional approaches include the utilization of specialized processing hardware such as Graphics Processing Units (GPUs) [14,15]. A widespread practice is to split the training data and use a number of SVMs to process the individual data chunks. This approach typically splits the training data set into a number of smaller fragments. This in turn reduces the individual training set and the overall training time. Most forms of decomposition which are based on a data splitting strategy approach tend to suffer from issues including convergence and accuracy. Challenges related to chunk aliasing, outlier accumulation and data imbalance/distribution tend to intensify problems in a parallel SVM context. This in turn impacts generalization, accuracy and convergence. Techniques such as random sampling have also been shown to exhibit similar accuracy challenges because of the induced probability distribution variance [16]. On the other hand, selective sampling techniques applied to SVMs try to sample the training data intelligently to maximize the performance of SVMs. However these normally require many scans of the entire data set which incurs high overhead in computation.

Abstract

Spam continues to inflict increased damage. Varying approaches including Support Vector Machine (SVM) based techniques have been proposed for spam classification. However, SVM training is a computationally intensive process. This paper presents a parallel SVM algorithm for scalable spam filtering. By distributing, processing and optimizing the subsets of the training data across multiple participating nodes, the distributed SVM reduces the training time significantly. Ontology based concepts are also employed to minimize the impact of accuracy degradation when distributing the training data amongst the SVM classifiers.

INTRODUCTION

Support Vector Machine (SVM) based approaches have persistently gained popularity in terms of their application for text classification and machine learning [1], [2]. Classification in SVM based approaches is founded on the notion of hyperplanes [3]. The hyperplanes act as class segregators in common binary classification, such as spam or ham in the context of spam filtering. SVM training is a computationally intensive process. Numerous SVM formulations, solvers and architectures for improving SVM performance have been explored and proposed [4], [5] including distributed and parallel computing techniques. SVM decomposition is another widespread technique for improving the performance in SVM training [6], [7]. Decomposition approaches work on the basis of identifying a small number of optimization variables and tackling a set of fixed size problems. Another widespread and effective practice is to split the training data into smaller fragments and use a number of SVM’s to process the individual data chunks. This in turn reduces overall training time. Various forms of summarizations and aggregations are then performed to process the final set of global support vectors [8]. Numerous forms of decomposition which are based on a data splitting strategy approach can suffer from issues including convergence and accuracy. Challenges related to chunk aliasing as well as outlier accumulation tend to intensify problems in a distributed SVM context. Adopting a training data set splitting strategy commonly amplifies issues related to data imbalance and data distribution instability.

Cyber Physical System (CPS) can be viewed as a large networked system, distributed geographically with each CPS site having hybrid computing patterns. Because of the collective might of CPS nodes, the capabilities of CPS could have far reaching effects in finding solutions to grand challenges in fields that combine science, engineering, and economics. Thus, CPS applications will require resources from different domains. In the absence of central authority to moderate and manage resource requests for CPS services, there is a need to facilitate the coordination of services and cooperation to share resource knowledge among resource providing nodes in CPS. There is a large variety of heterogeneous data stored in CPS nodes, includes traditional relation database that from different software vendors, including different types of NoSQL database, and different types of file, all of the data resources grow rapidly, become a great repository of data. For meeting the requirements of knowledge services, we need to extract the knowledge implied in heterogeneous data, represent and integrate the knowledge to support knowledge system. Knowledge Base Construction (KBC), the process of populating a knowledge base with semantic information from heterogeneous data, is an important and difficult task. In recent years, several large-scale knowledge bases have been constructed, such as YAGO[1], NELL[2], DBpedia[3], IBM’s Watson[4] and Microsoft’s EntityCube[5]. However, the rapid growth of huge heterogeneous textual data, stored in different data sources with different format and structure (e.g. mobile computing, social computing, Internet of Things, etc.), make it urgent to find a way to acquire and management knowledge from these data quickly. It is clear that building a distributed parallel framework to extract knowledge and construct knowledge base meet this requirement efficiently. Simultaneously, the present expression of heterogeneous textual data is weak semantic for random distribution so that it is not conducive to Parallel and Distributed Computing (PDC). It is obvious that there is a demand for a good data representation method with strong semantic can help parallel extraction of knowledge for better performance.

In recent decades, increasingly large amounts of data are generated from a variety of sources. The size of generated data per day on the Internet has already exceeded two exabytes [1]. Within one minute, 72 h of videos are uploaded to Youtube, around 30.000 new posts are created on the Tumblr blog platform, more than 100.000 Tweets are shared on Twitter and more than 200.000 pictures are posted on Facebook [1]. Big Data problems lead to several research questions such as (1) how to design scalable environments, (2) how to provide fault tolerance and (3) how to design efficient solutions. Most existing tools for storage, processing and analysis of data are inadequate for massive volumes of heterogeneous data. Consequently, there is an urgent need for more advanced and adequate Big Data solutions. Many definitions of Big Data have been proposed throughout the literature. Most of them agreed that Big Data problems share four main characteristics, referred to as the four V’s (Volume, Variety, Veracity and Velocity) [2]. The volume refers to the size of available datasets which typically require distributed storage and processing. The variety refers to the fact that Big Data is composed of several different types of data such as text, sound, image and video. The veracity refers to the biases, noise and abnormality in data. The velocity deals with the place at which data flows in from various sources like social networks, mobile devices and Internet of Things (IoT). In this paper, we first give an overview of most popular and widely used Big Data frameworks which are designed to cope with the above mentioned Big Data problems. We identify some key features which characterize Big Data frameworks. These key features include the programming model and the capability to allow for iterative processing of (streaming) data. We also give a categorization of existing frameworks according to the presented key features. Then, we present an experimental study on Big Data processing systems with several representative batch, stream and iterative workloads. Extensive surveys have been conducted to discuss Big Data Frameworks [3,4,5]. However, our experimental survey differs from existing ones by the fact that it considers performance evaluation of popular Big Data frameworks from different aspects. In our work, we compare the studied frameworks in the case of both batch processing and stream processing which is not studied in existing surveys. We also mention that our experimental study is concluded by some best practices related to the usage of the studied frameworks in several application domains. More specifically, the contributions of this paper are the following: • We present an overview of most popular Big Data frameworks and we categorize them according to some features. • We experimentally evaluate the performance of the presented frameworks and we present a comparative study of them in the case of both batch processing, stream processing. • We highlight best practices related to the use of popular Big Data frameworks in several application domains. The remainder of the paper is organized as follows. In Section 2, we present existing surveys on Big Data frameworks and we highlight the motivation of our work. In Section 3, we discuss existing Big Data frameworks and provide a categorization of them. In Section 4, we present a comparative study of the presented Big Data frameworks and we discuss the obtained results. In Section 5, we present some best practices of the studied frameworks. Some concluding points are given in Section 6.

استخراج با داده های بزرگ یا استخراج داده های عظیم به حوزه ی تحقیقاتی فعالی تبدیل شده است. استفاده از روش های رایج و ابزار نرم افزار داده کاوی برای اینکه یک کامپیوتر شخصی بتواند به طور موثر با پایگاه داده های بسیار بزرگ سر و کار داشته باشد، بسیار دشوار است. سکوهای رایانش موازی و ابری به عنوان راه حل بهتری برای استخراج داده های عظیم در نظر گرفته می شوند. مفهوم رایانش موازی بر مبنای تقسیم کردن یک مشکل بزرگ به قسمت های کوچک است و هر یک از این قسمت ها توسط یک پردازنده به طور مجزا انجام می شود. بعلاوه، این فرایندها به طور همزمان در روشی توزیعی و موازی انجام می شوند. دو روش رایج برای حل کردن این مشکل داده های بزرگ وجود دارد. مورد نخست رویه ی توزیعی بر مبنای الگوی موازی سازی داده هاست که یک مجموعه داده بزرگ می تواند به صورت دستی به n زیرمجموعه تقسیم شود و n الگوریتم برای هر n زیرمجموعه اجرا می گردد. نتیجه نهایی می تواند از ترکیبی از خروجی های تولید شده توسط n الگوریتم بدست آید. مورد دوم روند مبتنی بر نگاشت کاهش (MapReduce) در سکوی رایانش ابری است. این روند از فرایندهای نگاشت و کاهش تشکل شده است که مورد قبلی فیلتر کردن و طبقه بندی را انجام می دهد و مورد بعدی عملیات خلاصه را به منظور ایجاد نتیجه نهایی اجرا می کند. در این مقاله، هدف ما مقایسه ی تفاوت های عملکردی بین روش های توزیعی و نگاشت کاهش در پایگاه داده هایی با مقیاس بزرگ در قالب دقت و کارایی است. آزمایشات بر مبنای چهار پایگاه داده با مقیاس بزرگ است که برای مشکلات طبقه بندی داده ها مورد استفاده قرار می گیرند. نتایج حاکی از آن است که عملکردهای طبقه بندی روند مبتنی بر نگاشت کاهش بسیار پایدار هستند و مهم نیست که چند گره کامپیوتر مورد استفاده قرار می گیرد، و بهتر از ماشین منفرد خط مبنا و روندهای توزیعی جز برای پایگاه داده های عدم تعادل طبقه عمل می کند. بعلاوه، روند نگاشت کاهش نیازمند حداقل هزینه محاسباتی برای پردازش مجموعه داده های بزرگ است.