Mining Big Data for Frequent Patterns using MapReduce Computing

The main motivation of frequent pattern mining is to extract useful patterns from the data sets. Interesting associations among the data can be discovered by mining the frequent patterns. Among the different kinds of pattern mining, frequent itemset mining has been applied widely in many applications such as market basket analysis, medical applications, online transactions, social network analysis and so forth. An itemset is called frequent if the set of items in it appear frequently together. However, frequent itemset mining can find only the frequent itemsets, the time regularity of the items cannot be found. Sequential pattern mining considers both the frequency of the items and the order of items based on their time stamps. It attracted great deal of attention in many applications such as customer buying trend analysis, web access mining, natural disaster analysis and so forth. The patterns mined from sequential pattern mining algorithms do not consider the cost or profit of the item. A sequence that is not frequent in a dataset may contribute much to the overall profit of the organization due to its high profit. Hence, utility sequential pattern mining considers quantity and timestamp of items as well as profit of each item. Because of constantly arriving new data, the resultant patterns of frequent pattern mining may become obsolete over time. Hence, it is necessary to incrementally process the data in order to refresh the mining results without mining from scratch. The advancement in technology led to the generation of huge volumes of data from multiple sources such as social media, online transactions, internet applications and so forth. This era of big data pose a challenge to explore large volumes of data and extract the knowledge in the form of useful patterns. Moreover, the conventional methods used in mining patterns are not suitable for handling the big data. Hence, in this thesis, we investigate the solutions for frequent pattern mining on big data using a popular programming model known as MapReduce. Firstly, we propose a parallel algorithm for compressing the transactional data that makes the data simple and Bit Vector Product algorithm is proposed to mine the frequent itemsets from the compressed data. Secondly, distributed algorithm for mining sequential patterns using cooccurrence information is proposed. Here, we make use of item co-occurrence information and reduce the search space using the pruning strategies. Thirdly, distributed high utility time interval sequential patterns with time information between the successive items are mined. Finally, an incremental algorithm is proposed to make use of the knowledge obtained in ii previous mining while mining sequential patterns. All the proposed algorithms are tested on our in house Hadoop cluster composed of one master node and eight data nodes.