Health & Environmental Research Online (HERO)


Print Feedback Export to File
2839872 
Journal Article 
NO2: Speeding up Parallel Processing of Massive Compute-Intensive Tasks 
Wu, Y; Guo, W; Ren, J; Zhao, Xun; Zheng, W 
2014 
Yes 
I E E E Transactions on Computers
ISSN: 0018-9340 
63 
10 
2487-2499 
WOS:000343886800009 
Large-scale computing frameworks, either tenanted on the cloud or deployed in the high-end local cluster, have become an indispensable software infrastructure to support numerous enterprise and scientific applications. Tasks executed on these frameworks are generally classified into data-intensive and compute-intensive ones. However, most existing frameworks, led by MapReduce, are mainly suitable for data-intensive tasks. Their task schedulers assume that the proportion of data I/O reflects the task progress and state. Unfortunately, this assumption does not apply to most compute-intensive tasks. Due to biased estimation of task progress, traditional frameworks cannot timely cut off outliers and therefore largely prolong execution time when performing compute-intensive tasks. We propose a new framework designed for compute-intensive tasks. By using instrumentation and automatic instrument point selector, our framework estimates the compute-intensive task progress without resorting to data I/O. We employ a clustering method to identify outliers at runtime and perform speculative execution/aborting, speeding up task execution by up to 25%. Moreover, our improvement to bare instrumentation limits overhead within 0.1%, and the aborting-based execution only introduces 10% more average CPU usage. Low overhead and resource consumption make our framework practically usable in the production environment. 
Distributed system; parallel processing; compute-intensive; resource management