A heuristic approach for single path pipeline execution on grid

Abstract

In pipeline computation, a task is divided into a sequence of stages executed concurrently on different computing resources. When running on Grid, the cost of pipeline computation depends not only on the computation time of each stage, but the communication time to send intermediate data from one stage to another as well. Thus, such stages must be placed on nodes with proper computing resources and the data transferred between successive stages must be on the highest bandwidth in order to achieve the maximum throughput. Obviously, finding the optimal placement of pipeline stages on Grid nodes would not be easy. In this paper, we propose a heuristic algorithm called ThroughputPipeline to find the best pipeline placement on Grid that gives the maximum throughput for the single path pipeline execution where the flow of jobs cannot be split due to data dependency constraint. Our approach considers simultaneously both pipeline partitioning and the data transfer path. The experiment running in a real Grid environment gives higher throughput than other traditional placement algorithms. © 2008 IEEE.