.jpg)
在centos上配置hdfs（hadoop distributed file system）高可用性，需要遵循以下步驟：
1. 準(zhǔn)備工作
- 硬件準(zhǔn)備：確保至少有三臺服務(wù)器，用于部署NameNode和Secondary NameNode。
- 軟件準(zhǔn)備：安裝Java和Hadoop。
2. 安裝Java
sudo yum install java-1.8.0-openjdk-devel
3. 下載并解壓Hadoop
wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz tar -xzvf hadoop-3.3.1.tar.gz -C /usr/local/
4. 配置Hadoop
編輯/usr/local/hadoop/etc/hadoop/hadoop-env.sh文件，設(shè)置Java路徑：
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
5. 配置core-site.xml
<<span>configuration></span> <<span>property></span> <<span>name></span>fs.defaultFS</<span>name></span> <<span>value></span>hdfs://mycluster</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>hadoop.tmp.dir</<span>name></span> <<span>value></span>/usr/local/hadoop/tmp</<span>value></span> </<span>property></span> </<span>configuration></span>
6. 配置hdfs-site.xml
<<span>configuration></span> <<span>property></span> <<span>name></span>dfs.replication</<span>name></span> <<span>value></span>3</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.namenode.name.dir</<span>name></span> <<span>value></span>/usr/local/hadoop/data/namenode</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.datanode.data.dir</<span>name></span> <<span>value></span>/usr/local/hadoop/data/datanode</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.namenode.secondary.http-address</<span>name></span> <<span>value></span>secondarynamenode:50090</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.client.failover.proxy.provider.mycluster</<span>name></span> <<span>value></span>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.ha.fencing.methods</<span>name></span> <<span>value></span>sshfence</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.ha.fencing.ssh.private-key-files</<span>name></span> <<span>value></span>/home/hadoop/.ssh/id_rsa</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.namenode.shared.edits.dir</<span>name></span> <<span>value></span>qjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/mycluster</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>dfs.journalnode.edits.dir</<span>name></span> <<span>value></span>/usr/local/hadoop/data/journalnode</<span>value></span> </<span>property></span> </<span>configuration></span>
7. 配置yarn-site.xml
<<span>configuration></span> <<span>property></span> <<span>name></span>yarn.resourcemanager.ha.enabled</<span>name></span> <<span>value></span>true</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>yarn.resourcemanager.cluster-id</<span>name></span> <<span>value></span>yarn-cluster</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>yarn.resourcemanager.ha.rm-ids</<span>name></span> <<span>value></span>rm1,rm2</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>yarn.resourcemanager.hostname.rm1</<span>name></span> <<span>value></span>resourcemanager1</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>yarn.resourcemanager.hostname.rm2</<span>name></span> <<span>value></span>resourcemanager2</<span>value></span> </<span>property></span> <<span>property></span> <<span>name></span>yarn.resourcemanager.zk-address</<span>name></span> <<span>value></span>zookeeper1:2181,zookeeper2:2181,zookeeper3:2181</<span>value></span> </<span>property></span> </<span>configuration></span>
8. 配置mapred-site.xml
<<span>configuration></span> <<span>property></span> <<span>name></span>mapreduce.framework.name</<span>name></span> <<span>value></span>yarn</<span>value></span> </<span>property></span> </<span>configuration></span>
9. 啟動JournalNode
在所有journalnode服務(wù)器上執(zhí)行：
/usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
10. 格式化NameNode
在其中一個NameNode服務(wù)器上執(zhí)行：
/usr/local/hadoop/bin/hdfs namenode -format
11. 啟動Secondary NameNode
在Secondary NameNode服務(wù)器上執(zhí)行：
/usr/local/hadoop/sbin/hadoop-daemon.sh start secondarynamenode
12. 啟動NameNode
在第一個NameNode服務(wù)器上執(zhí)行：
/usr/local/hadoop/sbin/hadoop-daemon.sh start namenode
在第二個NameNode服務(wù)器上執(zhí)行：
/usr/local/hadoop/sbin/hadoop-daemon.sh start namenode
13. 啟動ResourceManager
在ResourceManager1服務(wù)器上執(zhí)行：
/usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
在ResourceManager2服務(wù)器上執(zhí)行：
/usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
14. 啟動DataNode
在所有DataNode服務(wù)器上執(zhí)行：
/usr/local/hadoop/sbin/hadoop-daemon.sh start datanode
15. 驗證配置
使用hdfs dfsadmin -report命令檢查集群狀態(tài)。
注意事項
- 確保所有節(jié)點之間的網(wǎng)絡(luò)通信正常。
- 確保防火墻允許必要的端口通信。
- 定期備份配置文件和數(shù)據(jù)。
通過以上步驟，你可以在centos上配置一個高可用的HDFS集群。
? 版權(quán)聲明
文章版權(quán)歸作者所有，未經(jīng)允許請勿轉(zhuǎn)載。
THE END
.png)
