生产中Hadoop分布式集群中可能存在着单点故障问题，如果Namenode宕机或是软硬件升级，集群将无法使用，所以进行搭建高可用的来消除单点故障。
Hadoop集群一般为一个NameNode和ResourceManager，但在实际生产环境中，若恰好具有NameNode和ResourceManager的节点出现故障，那么整个Hadoop集群将会崩溃，这是因为在HDFS中NameNode是系统的核心节点，ResourceManager负责整个系统的资源管理和分配。
为了解决单点故障问题，在Hadoop2后中引入了高可用机制，支持NameNode和ResourceManager一个主节点和一个备用节点，而在Hadoop3中继续对其进行了优化和提升，它支持一个主节点和多个备用节点。所谓高可用（High Availability, HA）就是7*24 小时不中断服务，消除单点故障。
Hadoop HA严格来说应该分成各个组件的HA机制：HDFS的HA和YARN的HA，可以通过配置多个NameNode和ResourceManager(Active/Standby)实现在集群中的热备来解决上述问题。
环境准备：
图片
节点相关进程如下：
图片
操作系统：centos8
内存：4G
Java 版本：jdk8
官网https://hadoop.apache.org/下载hadoop 3.3.0安装包解压至/usr/local下3台机器需修改的配置文件目录为/usr/local/hadoop/etc/hadoop下。
export JAVA_HOME=/usr/local/jdk#配置jdk路径#添加两行export HDFS_JOURNALNODE_USER=root export HDFS_ZKFC_USER=root
<?xml versinotallow="1.0"encoding="UTF-8"?><?xml-stylesheettype="text/xsl"href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration><!-- hdfs分布式文件系统名字/地址 --><property><name>fs.defaultFS</name><value>hdfs://ns</value></property><!--存放namenode、datanode数据的根路径 --><property><name>hadoop.tmp.dir</name><value>/usr/local/hadoop/tmp</value></property><!-- 存放journalnode数据的地址 --><property><name>dfs.journalnode.edits.dir</name><value>/usr/local/hadoop/tmp/jn</value></property><!-- 列出运行 ZooKeeper 服务的主机端口对 --><property><name>ha.zookeeper.quorum</name><value>hadoop:2181,k8s-2:2181,k8s-3:2181</value></property></configuration>
<?xml versinotallow="1.0"encoding="UTF-8"?><?xml-stylesheettype="text/xsl"href="configuration.xsl"?><!--Licensed under the Apache License,Version2.0(the"License");you maynotusethisfileexceptincompliancewiththe License.You may obtain a copyofthe License at http://www.apache.org/licenses/LICENSE-2.0Unless requiredbyapplicable laworagreedtoinwriting,softwaredistributedunder the Licenseisdistributedonan"AS IS"BASIS,WITHOUT WARRANTIESORCONDITIONSOFANYKIND,either expressorimplied.See the Licenseforthe specificlanguagegoverning permissionsandlimitations under the License.See accompanying LICENSEfile.--><configuration><property><name>dfs.replication</name><value>3</value></property><property><name>dfs.namenode.name.dir</name><value>/usr/local/hadoop/dfs/name</value></property><property><name>dfs.datanode.data.dir</name><value>/usr/local/hadoop/dfs/data</value></property><property><name>dfs.journalnode.edits.dir</name><value>/usr/local/hadoop/dfs/journalnode</value><description>The pathwherethe JournalNode daemon will store itslocalstate.</description></property><property><name>dfs.nameservices</name><value>ns</value><description>The logical nameforthis new nameservice.</description></property><property><name>dfs.ha.namenodes.ns</name><value>nn1,nn2,nn3</value><description>Uniqueidentifiersforeach NameNodeinthe nameservice.</description></property><property><name>dfs.namenode.rpc-address.ns.nn1</name><value>hadoop:8020</value><description>The fully-qualified RPC addressfornn1tolistenon.</description></property><property><name>dfs.namenode.rpc-address.ns.nn2</name><value>k8s-2:8020</value><description>The fully-qualified RPC addressfornn2tolistenon.</description></property><property><name>dfs.namenode.rpc-address.ns.nn3</name><value>k8s-3:8020</value><description>The fully-qualified RPC addressfornn3tolistenon.</description></property><property><name>dfs.namenode.http-address.ns.nn1</name><value>hadoop:9870</value><description>The fully-qualified HTTP addressfornn1tolistenon.</description></property><property><name>dfs.namenode.http-address.ns.nn2</name><value>k8s-2:9870</value><description>The fully-qualified HTTP addressfornn2tolistenon.</description></property><property><name>dfs.namenode.http-address.ns.nn3</name><value>k8s-3:9870</value><description>The fully-qualified HTTP addressfornn3tolistenon.</description></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://hadoop:8485;k8s-3:8485;k8s-2:8485/ns</value><description>The URI which identifies thegroupofJNswherethe NameNodes willwrite/readedits.</description></property><property><name>dfs.client.failover.proxy.provider.ns</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value><description>The Java class that HDFS clientsusetocontact the Active NameNode.</description></property><property><name>dfs.ha.fencing.methods</name><value>sshfence shell(/bin/true)</value><description>A listofscriptsorJava classes which will be usedtofence the Active NameNode during a failover.sshfence-SSHtothe Active NameNodeandkillthe process shell-run an arbitrary shell commandtofence the Active NameNode</description></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value><description>SetSSH privatekeyfile.</description></property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value><description>Automatic failover.</description></property></configuration>
<?xml versinotallow="1.0"?><?xml-stylesheettype="text/xsl"href="configuration.xsl"?><!--Licensed under the Apache License,Version2.0(the"License");you maynotusethisfileexceptincompliancewiththe License.You may obtain a copyofthe License at http://www.apache.org/licenses/LICENSE-2.0Unless requiredbyapplicable laworagreedtoinwriting,softwaredistributedunder the Licenseisdistributedonan"AS IS"BASIS,WITHOUT WARRANTIESORCONDITIONSOFANYKIND,either expressorimplied.See the Licenseforthe specificlanguagegoverning permissionsandlimitations under the License.See accompanying LICENSEfile.--><!-- Put site-specific property overrides in this file. --><configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>yarn.app.mapreduce.am.env</name><value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value></property><property><name>mapreduce.map.env</name><value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value></property><property><name>mapreduce.reduce.env</name><value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value></property></configuration>
<?xml versinotallow="1.0"?><!--Licensed under the Apache License,Version2.0(the"License");you maynotusethisfileexceptincompliancewiththe License.You may obtain a copyofthe License at http://www.apache.org/licenses/LICENSE-2.0Unless requiredbyapplicable laworagreedtoinwriting,softwaredistributedunder the Licenseisdistributedonan"AS IS"BASIS,WITHOUT WARRANTIESORCONDITIONSOFANYKIND,either expressorimplied.See the Licenseforthe specificlanguagegoverning permissionsandlimitations under the License.See accompanying LICENSEfile.--><configuration><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value><description>EnableRM HA.</description></property><property><name>yarn.resourcemanager.cluster-id</name><value>yrc</value><description>Identifies the cluster.</description></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2,rm3</value><description>Listoflogical IDsforthe RMs.e.g.,"rm1,rm2".</description></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>hadoop</value><description>Setrm1 service addresses.</description></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>k8s-2</value><description>Setrm2 service addresses.</description></property><property><name>yarn.resourcemanager.hostname.rm3</name><value>k8s-3</value><description>Setrm3 service addresses.</description></property><property><name>yarn.resourcemanager.webapp.address.rm1</name><value>hadoop:8088</value><description>Setrm1 web application addresses.</description></property><property><name>yarn.resourcemanager.webapp.address.rm2</name><value>k8s-2:8088</value><description>Setrm2 web application addresses.</description></property><property><name>yarn.resourcemanager.webapp.address.rm3</name><value>k8s-3:8088</value><description>Setrm3 web application addresses.</description></property><property><name>hadoop.zk.address</name><value>hadoop:2181,k8s-2:2181,k8s-3:2181</value><description>Addressofthe ZK-quorum.</description></property></configuration>
hadoop k8s-2k8s-3
通过https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.6.4/apache-zookeeper-3.6.4-bin.tar.gz 下载安装包，3台机器进行解压配置安装。
echo"1">/data/zookeeperdata/myid#不同机器id不同
zoo.cfg配置如下：
tickTime=2000initLimit=10syncLimit=5dataDir=/data/zookeeperdata#数据目录dataLogDir=/data/zookeeperdata/logs#日志目录clientPort=2181#端口server.1=192.xxx.xxx.128:2888:3888server.2=192.xxx.xxx.132:2888:3888server.3=192.xxx.xxx.131:2888:3888
vi /etc/profile
export JAVA_HOME=/usr/local/jdk export HAD00P_HOME=/usr/local/hadoop export PATH=$PATH:$HADOOP_HOME/bin export HADOOP_CLASSPATH=`hadoop classpath`
source/etc/profile
在所有节点上使用rm -rf /usr/local/hadoop/dfs命令，删除之前创建的存储路径，同时在master节点上执行mkdir -p /usr/local/hadoop/dfs/name /usr/local/hadoop/dfs/data /usr/local/hadoop/dfs/journalnode，再次创建存储路径。
在所有节点上使用rm -rf /usr/local/hadoop/tmp /usr/local/hadoop/logs && mkdir -p /usr/local/hadoop/tmp /usr/local/hadoop/logs命令，重置临时路径和日志信息。
通过以上步骤，Hadoop HA集群就配置完成了，当第一次启动HA集群时需要依次执行以下命令：
$ZOOKEEPER_HOME/bin/zkServer.shstart# 开启Zookeeper进程(所有节点上执行)$HADOOP_HOME/bin/hdfs--daemon start journalnode # 开启监控NameNode的管理日志的JournalNode进程(所有节点上执行)$HADOOP_HOME/bin/hdfs namenode-format# 命令格式化NameNode(在master节点上执行)scp-r/usr/local/hadoop/dfs k8s-2:/usr/local/hadoop# 将格式化后的目录复制到slave1中(在master节点上执行)scp-r/usr/local/hadoop/dfs k8s-3:/usr/local/hadoop# 将格式化后的目录复制到slave2中(在master节点上执行)$HADOOP_HOME/bin/hdfs zkfc-formatZK# 格式化Zookeeper Failover Controllers(在master节点上执行)start-dfs.sh&&start-yarn.sh# 启动HDFS和Yarn集群(在master节点上执行)
若不是第一次启动HA集群（常规启动HA集群），则只需要依次执行以下命令：
$ZOOKEEPER_HOME/bin/zkServer.shstart# 开启Zookeeper进程(所有节点上执行)start-all.sh或者$HADOOP_HOME/sbin/start-dfs.sh&&$HADOOP_HOME/sbin/start-yarn.sh# 启动HDFS和Yarn集群(在master节点上执行)
启动完成后每个节点使用jps命令会出现NameNode、DataNode、ResourceManager、NodeManager、JournalNode、DFSZKFailoverController、QuorumPeerMain和Jps 8个进程。
图片
图片
通过页面访问查看：
http://192.xxx.xxx.128:9870/
图片
http://192.xxx.xxx.128:8088/cluster/nodes
图片
图片
在开启HA集群并成功启动的情况下，在master节点中使用hdfs haadmin -getAllServiceState命令查看各个节点NameNode状态，接下来停止active状态节点的NameNode进程。
图片
Namenode active自动跳转其他节点，集群仍可用。
图片
图片
随后再启动该节点NameNode进程，最后再次查看状态，可以发现HDFS HA是正常的，并且没有发生抢占。
图片
在master节点中使用yarn rmadmin -getAllServiceState命令查看各个节点ResourceManager状态，接下来停止active状态节点的ResourceManage进程，ResourceManage active自动跳转到其他节点，集群仍可用，随后再启动该节点ResourceManager进程，最后再次查看状态，Failed状态恢复为standby。
图片
若要关闭集群可以master使用$HADOOP_HOME/sbin/stop-yarn.sh && $HADOOP_HOME/sbin/stop-dfs.sh命令或者stop-all.sh 即可关闭hadoop集群，然后关闭zookeeper,三台机器分别执行/data/apache-zookeeper-3.6.4-bin/bin/zkServer.sh stop。
