Streamed Sampling on Dynamic data as Support for Classification Model
Data mining process on dynamically changing data have several problems, such as unknown data size and changing of class distribution. Random sampling method commonly applied for extracting general synopsis from very large database. In this research, Vitter’s reservoir algorithm is used to retrieve k records of data from the database and put into the sample. Sample is used as input for classification task in data mining. Sample type is backing sample and it saved as table contains value of id, priority and timestamp. Priority indicates the probability of how long data retained in the sample. Kullback-Leibler divergence applied to measure the similarity between database and sample distribution. Result of this research is showed that continuously taken samples randomly is possible when transaction occurs. Kullback-Leibler divergence with interval from 0 to 0.0001, is a very good measure to maintain similar class distribution between database and sample. Sample results are always up to date on new transactions with similar class distribution. Classifier built from balance class distribution showed to have better performance than from imbalance one.
Braverman V, Ostrovsky R, Zainolo C. Optimal Sampling from Sliding Window. Proceedings of the twenty-eighth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems. 2009: 147-156.
Byung-Hoon P, George O, Nagiza F S. Sampling streaming data with replacement. Computational Statistics & Data Analysis. 2007; 52(2): 750-762.
Ferrandiz S, Boulle M. Supervised selection of dynamic features, with an application to tellecomunication data preparation. Proceedings of the 6th Industrial Conference on Data Mining conference on Advances in Data Mining: applications in Medicine, Web Mining, Marketing, Image and Signal Mining. 2006: 239-249.
Gemulla R, Lehner W. Sampling time-based sliding windows in bounded space. In Proc. of the 2008 ACMSIGMOD Intl. Conf. on Management of Data. 2008: 379–392.
Gibbons P B, Matias Y, Poosala V. Fast incremental maintenance of approximate Histograms. In Proc. VLDB. 1997: 466–475.