Introduction

Mining and visual analysis of contradictory

data using ConTra Dataset analysis and results

Performance evaluation of ConTra

Conclusion and the way forward

References

Abstract

Big datasets are often stored in fat fles and can contain contradictory data. Contradictory data undermines the soundness of the information from a noisy dataset. Traditional tools such as pie chart and bar chart are overwhelmed when used to visually identify contradictory data in multidimensional attribute-values of a big dataset. This work explains the importance of identifying contradictions in a noisy dataset. It also examines how contradictory data in a large and noisy dataset can be mined and visually analysed. The authors developed ‘ConTra’, an open source application which applies mutual exclusion rule in identifying contradictory data, existing in comma separated values (CSV) dataset. ConTra’s capability to enable the identifcation of contradictory data in diferent sizes of datasets is examined. The results show that ConTra can process large dataset when hosted in servers with fast processors. It is also shown in this work that ConTra is 100% accurate in identifying contradictory data of objects whose attribute values do not conform to the mutual exclusion rule of a dataset in CSV format. Diferent approaches through which ConTra can mine and identify contradictory data are also presented.

Introduction

A noisy dataset can contain contradictory data. Contradictory data is synonymous to incorrect data and it is important that such data be investigated and evaluated when analysing a noisy dataset. Diferent approaches to dealing with contradictory data have been proposed by diferent researchers. For example [1, 2] proposed methods for identifying and removing contradictory data in noisy datasets. However, the removal of contradictory data from a noisy dataset will increase the incompleteness in the dataset thereby reducing the soundness of any information from such set of data. It is therefore important to identify and evaluate contradictory instances when analysing a large and noisy dataset. Tis will improve the soundness of the analysis from such a dataset. Evidently, the analysis of big data is identifed as the next frontier for innovation and advancement of technology [3, 4]. Tere is therefore the need to identify appropriate approaches to dealing with contradictions in a large and noisy dataset. Tere are diferent forms of contradictions. For example, there are contradictions from the use of modal words, structural, subtle lexical contrasts, as well as world knowledge