Real time monitoring of system counters using Map Reduce Framework for effective analysis

Abstract

Several monitoring software’s provide flexibility to add various servers, performance system counters for real time monitoring. Application service managers use performance system counters to monitor servers and application health. It is constructive to detect anomalies earlier, take preventative actions and minimize the probability of failures of system to reduce uptime. The amount of data is huge for processing considering retention period and various global, process level system counters. The new approach, well known for offline processing, has been proposed for real-time data monitoring and analysis. The most notable solution that is proposed for managing and processing big data is the Map Reduce framework. Although Map-Reduce framework is very suitable for batch jobs but there is an increasing demand for non-batch processes on big data like: interactive jobs, real-time queries, and big data streams for faster and efficient monitoring.