Nchimbi Edward Pius and Liu Qin and Fion Yang and Zhu Hong Ming
Optimizing Hadoop Block Placement Policy and Cluster Blocks Distribution
1282 - 1288
2012
6
10
International Journal of Computer and Information Engineering
https://publications.waset.org/pdf/16836
https://publications.waset.org/vol/70
World Academy of Science, Engineering and Technology
The current Hadoop block placement policy do not fairly and evenly distributes replicas of blocks written to datanodes in a Hadoop cluster.
This paper presents a new solution that helps to keep the cluster in a balanced state while an HDFS client is writing data to a file in Hadoop cluster. The solution had been implemented, and test had been conducted to evaluate its contribution to Hadoop distributed file system.
It has been found that, the solution has lowered global execution time taken by Hadoop balancer to 22 percent. It also has been found that, Hadoop balancer respectively over replicate 1.75 and 3.3 percent of all redistributed blocks in the modified and original Hadoop clusters.
The feature that keeps the cluster in a balanced state works as a core part to Hadoop system and not just as a utility like traditional balancer. This is one of the significant achievements and uniqueness of the solution developed during the course of this research work.
Open Science Index 70, 2012