HelP: High-level Primitives For Large-Scale Graph Processing

Abstract

Large-scale graph processing systems typically expose a small set of functions, such as the compute() function of Pregel, or the gather(), apply(), and scatter() functions of PowerGraph. For some computations, these APIs are too low-level, yielding long and complex programs, but with shared coding patterns. Similar issues with the MapReduce framework have led to widely-used languages such as Pig Latin and Hive, which introduce higher-level primitives. We take an analogous approach for graph processing: we propose HelP, a set of high-level primitives that capture commonly appearing operations in large-scale graph computations. Using our primitives we have implemented a large suite of algorithms, some of which we previously implemented with the APIs of existing systems. Our experience has been that implementing algorithms using our primitives is more intuitive and much faster than using the APIs of existing distributed systems. All of our primitives and algorithms are fully implemented as a library on top of the open-source GraphX system.