Anomaly intrusion detection in big data environments calls for lightweight models that are able to achieve real-time performance during detection. Abstracting audit data provides a solution to improve the efficiency of data processing in intrusion detection. Data abstraction refers to abstract or extract the most relevant information from the massive dataset. In this work, we propose three strategies of data abstraction, namely, exemplar extraction, attribute selection and attribute abstraction. We first propose an effective method called exemplar extraction to extract representative subsets from the original massive data prior to building the detection models. Two clustering algorithms, Affinity Propagation (AP) and traditional k-means, are employed to find the exemplars from the audit data. K-Nearest Neighbor (k-NN), Principal Component Analysis (PCA) and one-class Support Vector Machine (SVM) are used for the detection. We then employ another two strategies, attribute selection and attribute extraction, to abstract audit data for anomaly intrusion detection. Two http streams collected from a real computing environment as well as the KDD’۹۹ benchmark data set are used to validate these three strategies of data abstraction. The comprehensive experimental results show that while all the three strategies improve the detection efficiency, the AP-based exemplar extraction achieves the best performance of data abstraction.

Introduction

The importance of computer network security is growing with the pervasive involvement of computers in people’s daily lives and in business processes within most organizations. As an important technique in the defense-indepth network security framework, intrusion detection has become a widely studied topic in computer networks in recent years. In general, the techniques for intrusion detection can be categorized as signature-based detection and anomaly detection. Signature-based detection (e.g., Snort [31]) relies on a database of signatures from known malicious threats. Anomaly detection, on the other hand, defines a profile of a subject’s normal activities and attempts to identify any unacceptable deviation as a potential attack. Typically, machine learning techniques are used to build normal profiles of a subject. Any observable behavior of a system, such as a network’s traffic [13,19], a computer host’s operating system [11,36] or a mobile application [2,39], can be used as the subject information. Anomaly detection has a potential to detect unforeseen attacks. As new attacks appear very frequently and signature-based detection methods may be overwhelmed by an abundance of polymorphic attacks, using anomaly detection sensors to discover zero-day attacks has become a necessity rather than an option [8]. We are entering the era of “big data” [۲۳]. The increasing volume of information generated by enterprises, the rise of social media and the Internet are fueling an exponential growth of data. Anomaly intrusion detection techniques are therefore challenged by the demand to process more massive data in higher dimensions at high speeds. A practical and efficient Intrusion Detection System (IDS) capable of detecting potential attacks is required so that resolutions can be implemented as quickly as possible.