Graph Processing On A Distributed System For A Massive Scale Using Epicg

Abstract

A large numbers of focused graph processing systems have been developed to cope with the increasing stipulate on graph analytics. Most of them require users to systematize a new framework in the cluster for graph processing and toggle to other systems to accomplish non-graph algorithms. This increases the involvedness of cluster management and results in unnecessary data transfer and redundancy. In this paper, we propose our graph processing engine, named as epiCG, which is built on top of epiC, an flexibility data processing system. The core of epiCG is a new unit called GraphUnit, which is able to not only perform iterative graph processing streamlined, but also collaborate with other types of units to accomplish any complex/multi-stage data analytics. The epiCG supports both edge-cut and vertex-cut partitioning methods, and for the latter method, we propose a novel light-weight greedy strategy that enables all the GraphUnits to generate vertex-cut partitioning in parallel. moreover, disparate existing graph processing systems, malfunction revitalization in epiCG is absolutely automatic. We compare epiCG with several graph processing systems via extensive experiments with real-life dataset and applications. The results show that epiCG possesses high orderliness and scalability, and performs tremendously well in large dataset settings, Proclaim its correctness for large-scale graph processing.