Hadoop\u662f\u4e00\u4e2a\u7531Apache\u57fa\u91d1\u4f1a\u6240\u5f00\u53d1\u7684\u5206\u5e03\u5f0f\u7cfb\u7edf\u57fa\u7840\u67b6\u6784\u3002\u7528\u6237\u53ef\u4ee5\u5728\u4e0d\u4e86\u89e3\u5206\u5e03\u5f0f\u5e95\u5c42\u7ec6\u8282\u7684\u60c5\u51b5\u4e0b，\u5f00\u53d1\u5206\u5e03\u5f0f\u7a0b\u5e8f\u3002 \u5145\u5206\u5229\u7528\u96c6\u7fa4\u7684\u5a01\u529b\u8fdb\u884c\u9ad8\u901f\u8fd0\u7b97\u548c\u5b58\u50a8\u3002<\/p>\n
HADOOP\u96c6\u7fa4\u5177\u4f53\u6765\u8bf4\u5305\u542b\u4e24\u4e2a\u96c6\u7fa4：HDFS\u96c6\u7fa4\u548cYARN\u96c6\u7fa4，\u4e24\u8005\u903b\u8f91\u4e0a\u5206\u79bb，\u4f46\u7269\u7406\u4e0a\u5e38\u5728\u4e00\u8d77\u3002 HDFS\u96c6\u7fa4\u8d1f\u8d23\u6d77\u91cf\u6570\u636e\u7684\u5b58\u50a8，\u96c6\u7fa4\u4e2d\u7684\u89d2\u8272\u4e3b\u8981\u6709：NameNode\u3001DataNode\u3001SecondaryNameNode YARN\u96c6\u7fa4\u8d1f\u8d23\u6d77\u91cf\u6570\u636e\u8fd0\u7b97\u65f6\u7684\u8d44\u6e90\u8c03\u5ea6，\u96c6\u7fa4\u4e2d\u7684\u89d2\u8272\u4e3b\u8981\u6709：ResourceManager\u3001NodeManager \u90a3mapreduce\u662f\u4ec0\u4e48\u5462？\u5b83\u5176\u5b9e\u662f\u4e00\u4e2a\u5206\u5e03\u5f0f\u8fd0\u7b97\u7f16\u7a0b\u6846\u67b6，\u662f\u5e94\u7528\u7a0b\u5e8f\u5f00\u53d1\u5305，\u7531\u7528\u6237\u6309\u7167\u7f16\u7a0b\u89c4\u8303\u8fdb\u884c\u7a0b\u5e8f\u5f00\u53d1，\u540e\u6253\u5305\u8fd0\u884c\u5728 HDFS\u96c6\u7fa4\u4e0a，\u5e76\u4e14\u53d7\u5230YARN\u96c6\u7fa4\u7684\u8d44\u6e90\u8c03\u5ea6\u7ba1\u7406\u3002<\/p>\n
\uf0b7HDFS（\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf）：\u89e3\u51b3\u6d77\u91cf\u6570\u636e\u5b58\u50a8 \uf0b7YARN（\u4f5c\u4e1a\u8c03\u5ea6\u548c\u96c6\u7fa4\u8d44\u6e90\u7ba1\u7406\u7684\u6846\u67b6）：\u89e3\u51b3\u8d44\u6e90\u4efb\u52a1\u8c03\u5ea6 \uf0b7MAPREDUCE（\u5206\u5e03\u5f0f\u8fd0\u7b97\u7f16\u7a0b\u6846\u67b6）：\u89e3\u51b3\u6d77\u91cf\u6570\u636e\u8ba1\u7b97<\/p>\n
zookeeper\u96c6\u7fa4\u9700\u8981\u81f3\u5c113\u4e2a\u8282\u70b9，\u5e76\u4e14\u8282\u70b9\u6570\u4e3a\u5947\u6570\u4e2a，\u53ef\u4ee5\u90e8\u7f72\u5728\u4efb\u610f\u72ec\u7acb\u8282\u70b9\u4e0a，NameNode\u53caResourceManager\u4f9d\u8d56zookeeper
\n\u8fdb\u884c\u4e3b\u5907\u9009\u4e3e\u548c\u5207\u6362<\/p>\n
NameNode\u81f3\u5c11\u9700\u89812\u4e2a\u8282\u70b9，\u4e00\u4e3b\u591a\u5907，\u53ef\u4ee5\u90e8\u7f72\u5728\u4efb\u610f\u72ec\u7acb\u8282\u70b9\u4e0a，\u7528\u4e8e\u7ba1\u7406HDFS\u7684\u540d\u79f0\u7a7a\u95f4\u548c\u6570\u636e\u5757\u6620\u5c04，
\n\u4f9d\u8d56zookeeper\u548czkfc\u5b9e\u73b0\u9ad8\u53ef\u7528\u548c\u81ea\u52a8\u6545\u969c\u8f6c\u79fb，\u5e76\u4e14\u4f9d\u8d56journalnode\u5b9e\u73b0\u72b6\u6001\u540c\u6b65<\/p>\n
ZKFailoverController\u5373zkfc，\u5728\u6240\u6709NameNode\u8282\u70b9\u4e0a\u542f\u52a8，\u7528\u4e8e\u76d1\u89c6\u548c\u7ba1\u7406NameNode\u72b6\u6001，\u53c2\u4e0e\u6545\u969c\u8f6c\u79fb<\/p>\n
DataNode\u81f3\u5c11\u9700\u89813\u4e2a\u8282\u70b9，\u56e0\u4e3ahdfs\u9ed8\u8ba4\u526f\u672c\u6570\u4e3a3，\u53ef\u4ee5\u90e8\u7f72\u5728\u4efb\u610f\u72ec\u7acb\u8282\u70b9\u4e0a，\u7528\u4e8e\u5b9e\u9645\u6570\u636e\u5b58\u50a8<\/p>\n
ResourceManager：\u81f3\u5c11\u9700\u89812\u4e2a\u8282\u70b9，\u4e00\u4e3b\u591a\u5907，\u53ef\u4ee5\u90e8\u7f72\u5728\u4efb\u610f\u72ec\u7acb\u8282\u70b9\u4e0a，\u4f9d\u8d56zookeeper\u5b9e\u73b0\u9ad8\u53ef\u7528\u548c\u81ea\u52a8\u6545\u969c\u8f6c\u79fb，
\n\u7528\u4e8e\u8d44\u6e90\u5206\u914d\u548c\u8c03\u5ea6<\/p>\n
NodeManager\u90e8\u7f72\u5728\u6240\u6709DataNode\u8282\u70b9\u4e0a，\u7528\u4e8e\u8282\u70b9\u8d44\u6e90\u7ba1\u7406\u548c\u76d1\u63a7<\/p>\n
journalnode\u81f3\u5c11\u9700\u89813\u4e2a\u8282\u70b9，\u5e76\u4e14\u8282\u70b9\u6570\u4e3a\u5947\u6570\u4e2a，\u53ef\u4ee5\u90e8\u7f72\u5728\u4efb\u610f\u72ec\u7acb\u8282\u70b9\u4e0a，\u7528\u4e8e\u4e3b\u5907NameNode\u72b6\u6001\u4fe1\u606f\u540c\u6b65<\/p>\n
| \u4e3b\u673a\u540d | IP\u5730\u5740 | \u7ec4\u4ef6 | | | | | | |
\n|—————-|————-|———-|———-|———-|————|—————–|———————|————-|
\n| idc-bigdata-01 | 10.62.1.181 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
\n| idc-bigdata-02 | 10.62.1.182 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
\n| idc-bigdata-03 | 10.62.1.183 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
\n| idc-bigdata-04 | 10.62.1.184 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |
\n| idc-bigdata-05 | 10.62.1.185 | zookeeper| NodeNode | DataNode | NodeManage | ResourceManager | ZKFailoverControlle | Journalnode |<\/p>\n
\u6240\u6709\u73af\u5883\u4f7f\u7528hadoop\u7528\u6237\u6765\u5b89\u88c5<\/p>\n
# uname -r
\n4.19.90-24.4.v2101.ky10.x86_64
\n\u9ed8\u8ba4\u5185\u6838\u4e3a4.19<\/p>\n
\u521b\u5efa \u7528\u6237
\nuseradd hadoop
\necho WLwl@2023|passwd –stdin hadoop <\/p>\n
visudo # \u53ef\u9009
\n\u6dfb\u52a0
\nhadoop ALL=(ALL) NOPASSWD: ALL <\/p>\n
\u5207\u6362\u65b9\u6cd5 su – hadoop<\/p>\n
hostnamectl set-hostname idc-bigdata-01 && hostname idc-bigdata-01 && hostname -f
\nhostnamectl set-hostname idc-bigdata-02 && hostname idc-bigdata-02 && hostname -f
\nhostnamectl set-hostname idc-bigdata-03 && hostname idc-bigdata-03 && hostname -f
\nhostnamectl set-hostname idc-bigdata-04 && hostname idc-bigdata-04 && hostname -f
\nhostnamectl set-hostname idc-bigdata-05 && hostname idc-bigdata-05 && hostname -f<\/p>\n
\u4e3a\u6bcf\u4e2a\u673a\u5668\u5355\u72ec\u914d\u7f6e\u4e3b\u673a\u540d，\u5386\u53f2\u8bb0\u5f55\u663e\u793a<\/p>\n
cat > \/etc\/profile.d\/sys.sh<<EOF
\nPS1='\\\\[\\\\e[32;1m\\\\][\\\\u@\\\\H \\\\W]\\\\\\\\$ \\\\[\\\\e[0m\\\\]'
\n#alias vi='vim'
\nalias grep='grep –color=auto'
\nalias ll='ls -l –color=auto –time-style=long-iso'
\nHISTSIZE=3000
\nHISTTIMEFORMAT="%Y-%m-%d %H:%M:%S –> "
\nEOF
\nsource \/etc\/profile.d\/sys.sh<\/p>\n
# \u96c6\u7fa4\u6240\u6709\u4e3b\u673a<\/p>\n
cat >\/etc\/hosts <<EOF
\n127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
\n::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
\n10.62.1.181 idc-bigdata-01
\n10.62.1.182 idc-bigdata-02
\n10.62.1.183 idc-bigdata-03
\n10.62.1.184 idc-bigdata-03
\n10.62.1.185 idc-bigdata-03
\nEOF<\/p>\n
for host in idc-bigdata-{01..05};do ping -c2 $host;done #\u9a8c\u8bc1<\/p>\n
# \u96c6\u7fa4\u6240\u6709\u4e3b\u673a
\nsystemctl disable –now firewalld && systemctl stop firewalld
\nsed -i 's\/^SELINUX=enforcing$\/SELINUX=disabled\/' \/etc\/selinux\/config && setenforce 0<\/p>\n
\u6dfb\u52a0\u5b9a\u65f6\u4efb\u52a1<\/p>\n
\u5982\u679c\u6ca1\u6709\u65f6\u95f4\u670d\u52a1\u5668\u9700\u8981\u81ea\u5df1\u5b89\u88c5，\u5e76\u6307\u5411\u65f6\u95f4\u670d\u52a1\u5668，\u96c6\u7fa4\u65f6\u95f4\u4e0d\u540c\u6b65\u4f1a\u5bfc\u81f4\u96c6\u7fa4\u8282\u70b9\u5b95\u673a<\/p>\n
yum install -y ntp \u6240\u6709\u8282\u70b9<\/p>\n
ntp\u670d\u52a1\u5668<\/p>\n
cat > \/etc\/ntp.conf << EOF<\/p>\n
# \u65e5\u5fd7\u914d\u7f6e
\nlogfile \/var\/log\/ntpd.log<\/p>\n
# \u914d\u7f6e\u65f6\u95f4\u670d\u52a1\u5668（\u963f\u91cc\u4e91）
\ndriftfile \/var\/lib\/ntp\/drift
\nrestrict default nomodify notrap nopeer noquery
\nrestrict 127.0.0.1
\nrestrict -6 ::1<\/p>\n
server ntp1.aliyun.com
\nserver ntp2.aliyun.com<\/p>\n
#\u5916\u90e8\u65f6\u95f4\u670d\u52a1\u5668\u4e0d\u53ef\u7528\u65f6，\u4ee5\u672c\u5730\u65f6\u95f4\u4f5c\u4e3a\u65f6\u95f4\u670d\u52a1
\nserver 127.127.1.0
\nfudge 127.127.1.0 stratum 10<\/p>\n
includefile \/etc\/ntp\/crypto\/pw
\nkeys \/etc\/ntp\/keys
\nEOF<\/p>\n
\u5ba2\u6237\u7aef<\/p>\n
cat > \/etc\/ntp.conf << EOF
\ndriftfile \/var\/lib\/ntp\/drift
\nrestrict default nomodify notrap nopeer noquery
\nrestrict 127.0.0.1
\nrestrict -6 ::1<\/p>\n
server 10.61.1.189<\/p>\n
includefile \/etc\/ntp\/crypto\/pw
\nkeys \/etc\/ntp\/keys
\nEOF<\/p>\n
systemctl restart ntpd && systemctl enable ntpd<\/p>\n
\u53ef\u9009 \u9ed8\u8ba4\u65f6\u533aAsia\/Shanghai
\ntimedatectl
\ntimedatectl set-timezone "Asia\/Shanghai"
\ntimedatectl set-ntp true<\/p>\n
ntpq -p<\/p>\n
\u5ba2\u6237\u7aef\u914d\u7f6e<\/p>\n
echo "*\/30 * * * * \/usr\/sbin\/ntpdate -u 10.62.1.191 >\/dev\/null 2>&1" > \/var\/spool\/cron\/root
\ncrontab -l<\/p>\n
\u6bcf\u53f0\u673a\u5668\u8bbe\u7f6e\u4ea4\u6362\u7a7a\u95f4
\n\u5efa\u8bae\u5c06\u4ea4\u6362\u7a7a\u95f4\u8bbe\u7f6e\u4e3a0，\u8fc7\u591a\u7684\u4ea4\u6362\u7a7a\u95f4\u4f1a\u5f15\u8d77GC\u8017\u65f6\u7684\u6fc0\u589e
\necho "vm.swappiness = 0" >> \/etc\/sysctl.conf && sysctl -p
\nsed -i '\/swap\/s\/^\\\\(.*\\\\)$\/#\\\\1\/g' \/etc\/fstab<\/p>\n
echo never > \/sys\/kernel\/mm\/transparent_hugepage\/enabled
\necho never > \/sys\/kernel\/mm\/transparent_hugepage\/defrag
\necho 'echo never > \/sys\/kernel\/mm\/transparent_hugepage\/defrag' >> \/etc\/rc.d\/rc.local
\necho 'echo never > \/sys\/kernel\/mm\/transparent_hugepage\/enabled' >> \/etc\/rc.d\/rc.local
\nchmod +x \/etc\/rc.d\/rc.local<\/p>\n
\u53ef\u9009
\nsed -i 's\/#UseDNS yes\/UseDNS no\/g' \/etc\/ssh\/sshd_config
\nsystemctl restart sshd<\/p>\n
\u8bbe\u7f6e StrictHostKeyChecking \u4e3ano<\/p>\n
sed -i 's\/^# StrictHostKeyChecking ask\/StrictHostKeyChecking no\/g' \/etc\/ssh\/ssh_config
\ngrep -w StrictHostKeyChecking \/etc\/ssh\/ssh_config<\/p>\n
sed -i '\/^*\/d' \/etc\/security\/limits.conf
\ncat >>\/etc\/security\/limits.conf <<EOF
\n* soft nproc 1024000
\n* hard nproc 1024000
\n* soft nofile 1024000
\n* hard nofile 1024000
\nEOF<\/p>\n
ulimit -n 1024000 && ulimit -u 1024000<\/p>\n
ulimit -n 65536 && ulimit -u 128605
\nulimit -a -H \u67e5\u770b<\/p>\n
\u8bf4\u660e：* \u4ee3\u8868\u9488\u5bf9\u6240\u6709\u7528\u6237
\nnoproc \u662f\u4ee3\u8868\u6700\u5927\u8fdb\u7a0b\u6570
\nnofile \u662f\u4ee3\u8868\u6700\u5927\u6587\u4ef6\u6253\u5f00\u6570
\n\u9700\u8981\u91cd\u8fde\u63a5\u624d\u751f\u6548<\/p>\n
cat >\/etc\/sysctl.d\/kernel.conf <<EOF<\/p>\n
# TCP Socket \u8bfb\u5199 Buffer \u8bbe\u7f6e:
\nnet.core.wmem_default = 8388608
\nnet.core.wmem_max = 16777216
\nnet.core.rmem_default = 8388608
\nnet.core.rmem_max = 16777216
\nnet.core.optmem_max = 16777216
\nnet.core.somaxconn = 65535
\nnet.core.netdev_max_backlog = 262144
\nnet.ipv4.tcp_rmem=4096 87380 16777216
\nnet.ipv4.tcp_wmem=4096 65536 16777216<\/p>\n
# \u542f\u7528IP\u8def\u7531\u8f6c\u53d1\u529f\u80fd
\nnet.ipv4.ip_forward = 1<\/p>\n
# \u5982\u679c\u5728\u8be5\u53c2\u6570\u6307\u5b9a\u7684\u79d2\u6570\u5185，TCP\u8fde\u63a5\u4e00\u76f4\u5904\u4e8e\u7a7a\u95f2，\u5219\u5185\u6838\u5f00\u59cb\u5411\u5ba2\u6237\u7aef\u53d1\u8d77\u5bf9\u5b83\u7684\u63a2\u6d4b，\u770b\u4ed6\u662f\u5426\u8fd8\u5b58\u6d3b\u7740
\nnet.ipv4.tcp_keepalive_time = 600<\/p>\n
# \u4ee5\u8be5\u53c2\u6570\u6307\u5b9a\u7684\u79d2\u6570\u4e3a\u65f6\u95f4\u95f4\u9694，\u5411\u5ba2\u6237\u7aef\u53d1\u8d77\u5bf9\u5b83\u7684\u63a2\u6d4b
\nnet.ipv4.tcp_keepalive_intvl = 30<\/p>\n
# \u5185\u6838\u53d1\u8d77\u5bf9\u5ba2\u6237\u7aef\u63a2\u6d4b\u7684\u6b21\u6570，\u5982\u679c\u90fd\u6ca1\u6709\u5f97\u5230\u76f8\u5e94，\u90a3\u4e48\u5c31\u65ad\u5b9a\u5ba2\u6237\u7aef\u4e0d\u53ef\u8fbe\u6216\u8005\u5df2\u5173\u95ed，\u5185\u6838\u5c31\u5173\u95ed\u8be5TCP\u8fde\u63a5，\u91ca\u653e\u76f8\u5173\u8d44\u6e90
\nnet.ipv4.tcp_keepalive_probes = 10<\/p>\n
# \u589e\u52a0\u7f51\u7edc\u7aef\u53e3\u8303\u56f4
\nnet.ipv4.ip_local_port_range = 1024 65000<\/p>\n
# \u589e\u52a0\u7f51\u7edcarp\u5730\u5740\u8868
\nnet.ipv4.neigh.default.gc_thresh1 = 1024
\nnet.ipv4.neigh.default.gc_thresh2 = 4096
\nnet.ipv4.neigh.default.gc_thresh3 = 8192<\/p>\n
# \u7981\u6b62TIME-WAIT\u72b6\u6001sockets\u7684\u5feb\u901f\u56de\u6536,\u542f\u7528\u4f1a\u5728NAT\u7f51\u7edc\u4e0b，\u4f1a\u5bfc\u81f4\u5927\u91cf\u7684TCP\u8fde\u63a5\u5efa\u7acb\u9519\u8bef\u3002
\nnet.ipv4.tcp_tw_recycle = 0<\/p>\n
# \u8868\u793a\u5f00\u542f\u91cd\u7528\u3002\u5141\u8bb8\u5c06TIME-WAIT sockets\u91cd\u65b0\u7528\u4e8e\u65b0\u7684TCP\u8fde\u63a5，\u9ed8\u8ba4\u4e3a0，\u8868\u793a\u5173\u95ed
\nnet.ipv4.tcp_tw_reuse = 1<\/p>\n
# nfs\u4f18\u5316
\nnet.ipv6.conf.all.disable_ipv6 = 1
\nnet.ipv6.conf.default.disable_ipv6 = 1<\/p>\n
# \u5173\u95edipv6\u534f\u8bae
\nnet.ipv6.conf.all.disable_ipv6 = 1<\/p>\n
# \u7981\u6b62\u4f7f\u7528swap\u7a7a\u95f4 \u53ea\u6709\u5f53\u7cfb\u7edf\u4e3a0\u65f6\u624d\u5141\u8bb8\u4f7f\u7528
\nvm.swappiness=0<\/p>\n
# \u4e0d\u68c0\u67e5\u7269\u7406\u5185\u5b58\u662f\u5426\u53ef\u7528
\n#vm.overcommit_memory = 1<\/p>\n
# \u5f00\u542f\u7f51\u6865\u6a21\u5f0f docker
\nnet.bridge.bridge-nf-call-ip6tables = 1
\nnet.bridge.bridge-nf-call-iptables = 1<\/p>\n
# linux \u6253\u5f00\u6587\u4ef6\u6570 too many open files \u89e3\u51b3\u65b9\u6cd5
\n# \u6ce8\u518c\u76d1\u542c\u76ee\u5f55\u7684\u6570\u91cf\u9650\u5236
\nfs.inotify.max_user_watches = 1280000
\n# \u8c03\u5927\u6700\u5927\u6587\u4ef6\u53e5\u67c4\u6570
\nfs.file-max = 52706963
\nfs.nr_open = 52700000<\/p>\n
# \u8bbe\u7f6e\u5185\u6838\u7f51\u7edcnetfilter\u6700\u5927\u8fde\u63a5\u6570 \u5185\u68384.0\u4ee5\u4e0a\u624d\u6709\u6b64\u6b64\u53c2\u6570
\n# \u6ce8\u610f\u5982\u679c\u9700\u8981\u542f\u7528\u6b64\u53c2\u6570，\u9700\u8981\u5f00\u542f\u7cfb\u7edf\u9632\u706b\u5899
\nnet.netfilter.nf_conntrack_max = 2310720
\nvm.max_map_count = 655360<\/p>\n
EOF<\/p>\n
sysctl -p \/etc\/sysctl.d\/kernel.conf <\/p>\n
\u6ce8：\u4f7f\u7528\u524d\u53bb\u9664\/etc\/sysctl.conf\u4e2d\u91cd\u590d\u914d\u7f6e<\/p>\n
\u6240\u6709NameNode\u8282\u70b9\u9700\u8981\u5bf9DataNode\u8282\u70b9\u514d\u5bc6，\u7528\u4e8eNameNode\u7ba1\u7406DataNode\u8fdb\u7a0b， \u6240\u6709NameNode\u8282\u70b9\u4e4b\u95f4\u4e5f\u9700\u8981\u76f8\u4e92\u514d\u5bc6，\u7528\u4e8e\u6545\u969c\u5207\u6362\u3002<\/p>\n
3\u4e2a\u8282\u70b9\u751f\u6210\u516c\u94a5\u548c\u79c1\u94a5，3\u4e2a\u8282\u70b9\u4e0a\u5206\u522b\u6267\u884c
\n\u5207\u6362\u7528\u6237，\u751f\u6210\u5bc6\u94a5\u5bf9<\/p>\n
su – hadoop
\nssh-keygen -t rsa -m PEM<\/p>\n
\u4e00\u76f4\u6309\u56de\u8f66，\u90fd\u8bbe\u7f6e\u4e3a\u9ed8\u8ba4\u503c，\u7136\u540e\u518d\u5f53\u524d\u7528\u6237\u7684Home\u76ee\u5f55\u4e0b\u7684.ssh\u76ee\u5f55\u4e2d\u4f1a\u751f\u6210\u516c\u94a5\u6587\u4ef6（id_rsa.pub）\u548c\u79c1\u94a5\u6587\u4ef6（id_rsa）<\/p>\n
\u6ce8\u610f：
\nssh-keygen -t rsa \u76f4\u63a5\u751f\u4ea7\u51fa\u6765\u7684\u683c\u5f0f\u662fOPENSSH，\u540e\u9762HDFS\u65e0\u6cd5\u5b9e\u73b0\u9ad8\u53ef\u7528\u81ea\u52a8\u5207\u6362\u3002，\u6240\u4ee5，\u9700\u8981\u52a0\u4e0a -m PEM
\n\u9519\u8bef\u8bb0\u5f55，\u8be6\u89c1 Hadoop \u4e4b \u9ad8\u53ef\u7528\u4e0d\u81ea\u52a8\u5207\u6362(ssh\u5bc6\u94a5\u65e0\u6548 Caused by: com.jcraft.jsch.JSchException: invalid privatekey )<\/p>\n
\u5206\u53d1\u811a\u672c \u6bcf\u53f0\u4e3b\u673a\u4e0a\u90fd\u6267\u884c
\nfor i in idc-bigdata-{01..05};do ssh-copy-id $i -o StrictHostKeyChecking=no ;done<\/p>\n
\u53e6\u5916\u4e24\u53f0\u673a\u5668\u6267\u884c\u540c\u6837\u64cd\u4f5c，\u751f\u6210\u516c\u94a5\u548c\u79c1\u94a5\u540e，\u5206\u53d1\u7ed9\u662f\u4e09\u53f0\u673a\u5668<\/p>\n
\u9a8c\u8bc1
\nfor host in idc-bigdata-{01..05}
\ndo
\necho ======print hostname=======
\nssh $host hostname
\necho ======print ip addr =========
\nssh $host ip a|awk -F "[ \/]+" '\/inet\/ && \/brd\/{print $3}'
\necho
\ndone<\/p>\n
su – hadoop
\nssh-keygen -t rsa -m PEM \u751f\u6210RSA \u7c7b\u578b<\/p>\n
5\u4e2a\u8282\u70b9\u5bf9idc-bigdata-01\u8282\u70b9\u514d\u5bc6，5\u4e2a\u8282\u70b9\u4e0a\u5206\u522b\u6267\u884c<\/p>\n
ssh-copy-id idc-bigdata-01<\/p>\n
idc-bigdata-01\u8282\u70b9\u6267\u884c，\u5206\u53d1authorized_keys\u6587\u4ef6\u5230\u5176\u4ed6\u8282\u70b9<\/p>\n
scp \/root\/.ssh\/authorized_keys idc-bigdata-02:\/root\/.ssh\/authorized_keys
\nscp \/root\/.ssh\/authorized_keys idc-bigdata-02:\/root\/.ssh\/authorized_keys<\/p>\n
idc-bigdata-01\u8282\u70b9\u6267\u884c，\u6240\u6709\u8282\u70b9\u516c\u94a5\u5199\u5165known_hosts
\nssh-keyscan -t ecdsa idc-bigdata-01 idc-bigdata-02 idc-bigdata-03 > \/root\/.ssh\/known_hosts<\/p>\n
idc-bigdata-01\u8282\u70b9\u6267\u884c，\u5206\u53d1known_hosts\u6587\u4ef6\u5230\u5176\u4ed6\u8282\u70b9<\/p>\n
scp \/root\/.ssh\/known_hosts idc-bigdata-02:\/root\/.ssh\/
\nscp \/root\/.ssh\/known_hosts idc-bigdata-03:\/root\/.ssh\/<\/p>\n
\u9a8c\u8bc1\u514d\u5bc6\u6210\u529f，\u4efb\u610fnamenode\u8282\u70b9ssh\u5230\u5176\u4ed6\u8282\u70b9\u80fd\u591f\u514d\u5bc6\u767b\u5f55
\nssh idc-bigdata-01
\nssh idc-bigdata-02
\nssh idc-bigdata-03<\/p>\n
\u6240\u6709\u8282\u70b9\u5b89\u88c5java，\u56e0\u4e3azookeeper\u53cahadoop\u4f9d\u8d56java\u73af\u5883\u3002<\/p>\n
\u5982\u679c\u4f7f\u7528open-jdk \u8fd9\u91cc\u53ef\u4ee5\u4e0d\u9700\u8981\u5b89\u88c5<\/p>\n
\u624b\u52a8\u5b89\u88c5\u6b65\u9aa4<\/p>\n
\u5378\u8f7dopen-jdk<\/p>\n
\u5982\u679c\u5df2\u5b89\u88c5open-jdk ，\u9700\u8981\u5378\u8f7dopen-jdk<\/p>\n
rpm -qa|grep java
\n[root@idc-bigdata-03 ~]# rpm -qa|grep java
\njavapackages-tools-5.3.0-2.ky10.noarch
\njavapackages-filesystem-5.3.0-2.ky10.noarch
\njava-11-openjdk-headless-11.0.9.11-4.ky10.x86_64
\njava-1.8.0-openjdk-headless-1.8.0.272.b10-7.ky10.x86_64
\njava-1.8.0-openjdk-1.8.0.272.b10-7.ky10.x86_64
\ntzdata-java-2020a-8.p01.ky10.noarch
\njava-11-openjdk-11.0.9.11-4.ky10.x86_64<\/p>\n
yum remove java-openjdk -y
\n[root@idc-bigdata-01 conf]# rpm -e –nodeps java-1.8.0-openjdk-headless-1.8.0.342.b07-0.p01.ky10.x86_64 java-11-openjdk-headless-11.0.9.11-4.ky10.x86_64 tzdata-java-2020a-8.p01.ky10.noarch<\/p>\n
[root@idc-bigdata-01 conf]# rpm -qa|grep java
\njavapackages-tools-5.3.0-2.ky10.noarch
\njavapackages-filesystem-5.3.0-2.ky10.noarch<\/p>\n
\u6216
\nrpm -qa|grep java|xargs -i rpm -e –nodeps {}<\/p>\n
\u5b89\u88c5jdk<\/p>\n
\u4e0b\u8f7djdk \u5ffd\u7565
\n\u4e0a\u4f20jdk-8u181-linux-x64.tar.gz \u5230\/opt
\nmkdir -p \/usr\/java
\ntar xf \/opt\/jdk-8u181-linux-x64.tar.gz -C \/usr\/java<\/p>\n
cat > \/etc\/profile.d\/jdk8.sh << EOF
\nexport JAVA_HOME=\/usr\/java\/jdk1.8.0_181
\nexport PATH=\\\\$PATH:\\\\$JAVA_HOME\/bin:\/usr\/local\/bin
\nexport CLASSPATH=\\\\$JAVA_HOME\/lib
\nEOF
\nsource \/etc\/profile.d\/jdk8.sh
\njava -version<\/p>\n
\u5176\u5b83\u4e3b\u673a\u5b89\u88c5 <\/p>\n
for host in idc-bigdata-{02..03}
\ndo
\n ssh $host mkdir -p \/usr\/java
\nscp -r \/usr\/java\/jdk1.8.0_181 $host:\/usr\/java
\ndone<\/p>\n
cat > \/etc\/profile.d\/jdk8.sh << EOF
\nexport JAVA_HOME=\/usr\/java\/jdk1.8.0_181
\nexport PATH=\\\\$PATH:\\\\$JAVA_HOME\/bin:\/usr\/local\/bin
\nexport CLASSPATH=\\\\$JAVA_HOME\/lib
\nEOF
\nsource \/etc\/profile.d\/jdk8.sh
\njava -version<\/p>\n
\u62a5\u9519<\/p>\n
ln -sf \/usr\/lib64\/libLLVM-7.0.0.so \/usr\/lib64\/libLLVM-7.so<\/p>\n
yum istall -y psmisc vim net-tools wget iotop iftop <\/p>\n
NameNode\u4e3b\u5907\u5207\u6362\u4f9d\u8d56fuser，\u6240\u6709nameNode\u8282\u70b9\u5b89\u88c5psmisc<\/p>\n
\u751f\u4ea7\u73af\u5883\u9700\u89813\u52305\u4e2a\u8282\u70b9\u505a\u96c6\u7fa4<\/p>\n
Zookeeper \u7248\u672c3.8<\/p>\n
\u4ee5\u4e0b\u64cd\u4f5c\u57283\u4e2a\u8282\u70b9\u4e0a\u6267\u884c\u3002<\/p>\n
\u6240\u6709\u8282\u70b9\u5b89\u88c5zookeeper<\/p>\n
\u6240\u6709\u8282\u70b9<\/p>\n
tar xf apache-zookeeper-3.8.0-bin.tar.gz -C \/usr\/local
\nln -sf \/usr\/local\/apache-zookeeper-3.8.0-bin \/usr\/local\/zookeeper
\nls -lh \/usr\/local\/<\/p>\n
scp -r \/usr\/local\/apache-zookeeper-3.8.0-bin idc-bigdata-02:\/usr\/local
\nscp -r \/usr\/local\/apache-zookeeper-3.8.0-bin idc-bigdata-03:\/usr\/local<\/p>\n
\u8f6f\u8fde\u63a5
\nln -sf \/usr\/local\/apache-zookeeper-3.8.0-bin \/usr\/local\/zookeeper<\/p>\n
for host in idc-bigdata-{02..03}
\ndo
\nscp -r apache-zookeeper-3.8.0-bin.tar.gz $host:\/usr\/local
\ndone<\/p>\n
\u6240\u6709\u8282\u70b9\u6267\u884c<\/p>\n
cat > \/etc\/profile.d\/zookeeper.sh <<EOF
\nexport ZOOKEEPER_HOME=\/usr\/local\/zookeeper
\nexport PATH=\\\\$ZOOKEEPER_HOME\/bin:\\\\$PATH
\nEOF
\nsource \/etc\/profile<\/p>\n
\u6240\u6709\u8282\u70b9\u6267\u884c<\/p>\n
cat >\/usr\/local\/zookeeper\/conf\/zoo.cfg<<EOF
\ntickTime=2000
\ninitLimit=10
\nsyncLimit=5<\/p>\n
#cluster
\nserver.1=10.62.1.181:2188:3888
\nserver.2=10.62.1.182:2188:3888
\nserver.3=10.62.1.183:2188:3888
\n#data dir
\ndataDir=\/data\/zookeeper\/data
\ndataLogDir=\/data\/zookeeper\/logs
\nclientPort=2181
\nmaxClientCnxns=60
\n# https:\/\/prometheus.io Metrics Exporter
\nmetricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
\nmetricsProvider.httpHost=0.0.0.0
\nmetricsProvider.httpPort=7000
\nmetricsProvider.exportJvmInfo=true
\nEOF<\/p>\n
3888\u662f\u96c6\u7fa4\u9009\u4e3e\u7b49\u4fe1\u606f\u901a\u544a\u7684\u7aef\u53e3\u3002
\n2188\u662f\u8fdb\u884c\u6570\u636e\u540c\u6b65\u7684\u7aef\u53e3
\n2181\u662f\u5ba2\u6237\u7aef\u8fde\u63a5\u670d\u52a1\u7aef\u7684\u7aef\u53e3<\/p>\n
\u521b\u5efamyid\u6587\u4ef6，id\u5728\u6574\u4f53\u4e2d\u5fc5\u987b\u662f\u552f\u4e00\u7684，\u5e76\u4e14\u5e94\u8be5\u5177\u67091\u5230255\u4e4b\u95f4\u7684\u503c， \u4e3b\u914d\u7f6e\u6587\u4ef6\u4e2d\u7684server id \u8981\u548c\u5176\u5f53\u524d\u4e3b\u8282\u70b9\u4e2d\u7684myid\u4fdd\u6301\u4e00\u81f4，\u5206\u522b\u57283\u4e2a\u8282\u70b9\u4e0a\u6267\u884c<\/p>\n
\u521b\u5efadata\u548clog\u76ee\u5f55： mkdir -p \/data\/zookeeper\/{data,logs}<\/p>\n
#idc-bigdata-01
\nmkdir -p \/data\/zookeeper\/data
\necho "1" > \/data\/zookeeper\/data\/myid <\/p>\n
#idc-bigdata-02
\nmkdir -p \/data\/zookeeper\/data
\necho "2" > \/data\/zookeeper\/data\/myid <\/p>\n
#idc-bigdata-03
\nmkdir -p \/data\/zookeeper\/data
\necho "3" > \/data\/zookeeper\/data\/myid <\/p>\n
#idc-bigdata-04
\nmkdir -p \/data\/zookeeper\/data
\necho "4" > \/data\/zookeeper\/data\/myid <\/p>\n
#idc-bigdata-05
\nmkdir -p \/data\/zookeeper\/data
\necho "5" > \/data\/zookeeper\/data\/myid <\/p>\n
\u521b\u5efazookeeper\u7528\u6237 \u53ef\u9009 \u8fd9\u91cc\u4f7f\u7528root\u542f\u52a8\u670d\u52a1 useradd -r -s \/bin\/false zookeeper chown -R zookeeper:zookeeper \/usr\/local\/zookeeper<\/p>\n
\u4f7f\u7528systemd\u7ba1\u7406zookeeper\u670d\u52a1<\/p>\n
cat > \/usr\/lib\/systemd\/system\/zookeeper.service << EOF
\n[Unit]
\nDescription=Zookeeper Service<\/p>\n
[Service]
\nType=simple
\nEnvironment="JAVA_HOME=\/usr\/java\/jdk1.8.0_181"
\nWorkingDirectory=\/usr\/local\/zookeeper\/
\nPIDFile=\/usr\/local\/zookeeper\/data\/zookeeper_server.pid
\nSyslogIdentifier=zookeeper
\nUser=root
\nGroup=root
\nExecStart=\/usr\/local\/zookeeper\/bin\/zkServer.sh start
\nExecStop=\/usr\/local\/zookeeper\/bin\/zkServer.sh stop
\nRestart=always
\nTimeoutSec=20
\nSuccessExitStatus=130 143
\nRestart=on-failure<\/p>\n
[Install]
\nWantedBy=multi-user.target
\nEOF<\/p>\n
\u542f\u52a8zookeeper\u670d\u52a1，\u5e76\u8bbe\u4e3a\u5f00\u673a\u542f\u52a8<\/p>\n
systemctl daemon-reload && systemctl enable zookeeper && systemctl restart zookeeper && systemctl status zookeeper<\/p>\n
#systemctl enable –now zookeeper<\/p>\n
\u67e5\u770bzookeeper\u4e3b\u5907\u72b6\u6001<\/p>\n
[hadoop@idc-bigdata-01 ~]$ xshell zkServer.sh status
\n==================xshell zkServer.sh status @idc-bigdata-01======================<\/p>\n
Authorized users only. All activities may be monitored and reported.
\nZooKeeper JMX enabled by default
\nUsing config: \/usr\/local\/zookeeper\/bin\/..\/conf\/zoo.cfg
\nClient port found: 2181. Client address: localhost. Client SSL: false.
\nMode: follower
\n==================xshell zkServer.sh status @idc-bigdata-02======================<\/p>\n
Authorized users only. All activities may be monitored and reported.
\nZooKeeper JMX enabled by default
\nUsing config: \/usr\/local\/zookeeper\/bin\/..\/conf\/zoo.cfg
\nClient port found: 2181. Client address: localhost. Client SSL: false.
\nMode: leader
\n==================xshell zkServer.sh status @idc-bigdata-03======================<\/p>\n
Authorized users only. All activities may be monitored and reported.
\nZooKeeper JMX enabled by default
\nUsing config: \/usr\/local\/zookeeper\/bin\/..\/conf\/zoo.cfg
\nClient port found: 2181. Client address: localhost. Client SSL: false.
\nMode: follower<\/p>\n
Hadoop\u5b89\u88c5\u914d\u7f6e \u5b98\u7f51\u4e0b\u8f7d\u5730\u5740：https:\/\/hadoop.apache.org\/releases.html<\/p>\n
\u5c06\u4e0b\u8f7d\u597d\u7684hadoop-3.3.5.tar.gz\u5305\u89e3\u538b\u5230 \/usr\/module\u4e0b<\/p>\n
mkdir -p \/opt\/software
\nchown -R hadoop:hadoop \/opt\/software
\nwget https:\/\/mirrors.aliyun.com\/apache\/hadoop\/common\/hadoop-3.3.5\/hadoop-3.3.5.tar.gz -P \/opt\/software<\/p>\n
mkdir -p \/usr\/module
\ntar xf \/opt\/software\/hadoop-3.3.5.tar.gz -C \/usr\/module
\nln -sf \/opt\/module\/hadoop-3.3.5 \/opt\/module\/hadoop
\nchown -R hadoop:hadoop \/opt\/module\/hadoop*<\/p>\n
\u6dfb\u52a0\u4ee5\u4e0b\u53c2\u6570<\/p>\n
vim \/etc\/profile.d\/myenv.sh<\/p>\n
# HADOOP_HOME
\nexport HADOOP_HOME=\/opt\/module\/hadoop-3.3.5
\nexport HADOOP_CONF_DIR=\/opt\/module\/hadoop-3.3.5\/etc\/hadoop
\nexport PATH=$PATH:$HADOOP_HOME\/bin
\nexport PATH=$PATH:$HADOOP_HOME\/sbin
\n# ZOO_HOME
\n#export ZOO_HOME=\/opt\/module\/apache-zookeeper-3.8.0-bin
\n#export PATH=$PATH:$ZOO_HOME\/bin
\n# HIVE_HOME
\nexport HIVE_HOME=\/opt\/module\/apache-hive-3.1.2-bin
\nexport PATH=$PATH:$HIVE_HOME\/bin
\n# SPARK_HOME
\n#export SPARK_HOME=\/opt\/module\/spark-3.1.3-bin-hadoop3.2
\nexport SPARK_HOME=\/opt\/module\/spark-3.3.2-bin-hadoop3
\nexport PATH=$PATH:$SPARK_HOME\/bin:$SPARK_HOME\/sbin
\n# KAFKA_HOME
\n#export KAFKA_HOME=\/opt\/module\/kafka_2.12-3.2.0
\n#export PATH=$PATH:$KAFKA_HOME\/bin
\n# HBASE_HOME
\n#export HBASE_HOME=\/opt\/module\/hbase-2.4.13
\nexport HBASE_HOME=\/opt\/module\/hbase-2.5.3
\n#export HBASE_HOME=\/opt\/module\/hbase-2.5.3-hadoop3
\nexport PATH=$PATH:$HBASE_HOME\/bin
\n# PHOENIX_HOME
\nexport PHOENIX_HOME=\/opt\/module\/phoenix-hbase-2.4-5.1.2-bin
\nexport PHOENIX_CLASSPATH=$PHOENIX_HOME
\nexport PATH=$PATH:$PHOENIX_HOME\/bin<\/p>\n
# JAVA_HOME
\n#export JAVA_HOME=\/opt\/module\/jdk1.8.0_181
\n#export PATH=$PATH:$JAVA_HOME\/bin<\/p>\n
# M2_HOME
\n#export M2_HOME=\/opt\/module\/maven-3.8.6
\n#export PATH=$PATH:$M2_HOME\/bin<\/p>\n
# EFAK HOME
\n#export KE_HOME=\/opt\/module\/efak-web-3.0.1
\n#export PATH=$PATH:$KE_HOME\/bin<\/p>\n
# FLINK_HOME
\n#export FLINK_HOME=\/opt\/module\/flink-1.13.6
\nexport FLINK_HOME=\/opt\/module\/flink-1.13.1
\nexport PATH=$PATH:$FLINK_HOME\/bin<\/p>\n
#KYUUBI_HOME
\nexport KYUUBI_HOME=\/opt\/module\/apache-kyuubi-1.7.1-bin
\nexport PATH=$PATH:$KYUUBI_HOME\/bin<\/p>\n
# HADOOP_CLASSPATH
\nexport HADOOP_CLASSPATH=$(hadoop classpath)<\/p>\n
\u52a0\u8f7d\u53d8\u91cf<\/p>\n
source \/etc\/profile.d\/myenv.sh<\/p>\n
\u67e5\u770b hadoop \u7248\u672c
\n[root@idc-bigdata-01 ~]# hdfs version<\/p>\n
\u73af\u5883\u53d8\u91cf\u6839\u636e\u9700\u8981\u4fee\u6539<\/p>\n
bin：Hadoop\u6700\u57fa\u672c\u7684\u7ba1\u7406\u811a\u672c\u548c\u4f7f\u7528\u811a\u672c\u7684\u76ee\u5f55，\u8fd9\u4e9b\u811a\u672c\u662fsbin\u76ee\u5f55\u4e0b\u7ba1\u7406\u811a\u672c\u7684\u57fa\u7840\u5b9e\u73b0，\u7528\u6237\u53ef\u4ee5\u76f4\u63a5\u4f7f\u7528\u8fd9\u4e9b\u811a\u672c\u7ba1\u7406\u548c\u4f7f\u7528Hadoop\u3002
\netc：Hadoop\u914d\u7f6e\u6587\u4ef6\u6240\u5728\u7684\u76ee\u5f55，\u5305\u62eccore-site.xml\u3001hdfs-site.xml\u3001mapred-site.xml\u7b49\u4eceHadoop1.0\u7ee7\u627f\u800c\u6765\u7684\u914d\u7f6e\u6587\u4ef6\u548cyarn-site.xml\u7b49Hadoop2.0\u65b0\u589e\u7684\u914d\u7f6e\u6587\u4ef6\u3002
\ninclude：\u5bf9\u5916\u63d0\u4f9b\u7684\u7f16\u7a0b\u5e93\u5934\u6587\u4ef6（\u5177\u4f53\u52a8\u6001\u5e93\u548c\u9759\u6001\u5e93\u5728lib\u76ee\u5f55\u4e2d），\u8fd9\u4e9b\u5934\u6587\u4ef6\u5747\u662f\u7528C++\u5b9a\u4e49\u7684，\u901a\u5e38\u7528\u4e8eC++\u7a0b\u5e8f\u8bbf\u95eeHDFS\u6216\u8005\u7f16\u5199MapReduce\u7a0b\u5e8f\u3002
\nlib：\u8be5\u76ee\u5f55\u5305\u542b\u4e86Hadoop\u5bf9\u5916\u63d0\u4f9b\u7684\u7f16\u7a0b\u52a8\u6001\u5e93\u548c\u9759\u6001\u5e93，\u4e0einclude\u76ee\u5f55\u4e2d\u7684\u5934\u6587\u4ef6\u7ed3\u5408\u4f7f\u7528\u3002
\nlibexec：\u5404\u4e2a\u670d\u52a1\u5bf9\u7528\u7684shell\u914d\u7f6e\u6587\u4ef6\u6240\u5728\u7684\u76ee\u5f55，\u53ef\u7528\u4e8e\u914d\u7f6e\u65e5\u5fd7\u8f93\u51fa\u3001\u542f\u52a8\u53c2\u6570（\u6bd4\u5982JVM\u53c2\u6570）\u7b49\u57fa\u672c\u4fe1\u606f\u3002
\nsbin：Hadoop\u7ba1\u7406\u811a\u672c\u6240\u5728\u7684\u76ee\u5f55，\u4e3b\u8981\u5305\u542bHDFS\u548cYARN\u4e2d\u5404\u7c7b\u670d\u52a1\u7684\u542f\u52a8\/\u5173\u95ed\u811a\u672c\u3002
\nshare：Hadoop\u5404\u4e2a\u6a21\u5757\u7f16\u8bd1\u540e\u7684jar\u5305\u6240\u5728\u7684\u76ee\u5f55，\u5b98\u65b9\u81ea\u5e26\u793a\u4f8b\u3002<\/p>\n
Hadoop\u914d\u7f6e\u6587\u4ef6\u4fee\u6539（\u6ce8\u610f：\u4ee5\u4e0b\u6240\u6709\u64cd\u4f5c\u90fd\u5728idc-bigdata-01\u4e3b\u673a\u8fdb\u884c） Hadoop\u5b89\u88c5\u4e3b\u8981\u5c31\u662f\u914d\u7f6e\u6587\u4ef6\u7684\u4fee\u6539，\u4e00\u822c\u5728\u4e3b\u8282\u70b9\u8fdb\u884c\u4fee\u6539，\u5b8c\u6bd5\u540escp\u4e0b\u53d1\u7ed9\u5176\u4ed6\u5404\u4e2a\u4ece\u8282\u70b9\u673a\u5668 Hadoop\u5b89\u88c5\u76ee\u5f55\u4e0b\u7684etc\/hadoop\u76ee\u5f55\u4e2d，\u9700\u4fee\u6539core-site.xml\u3001hdfs-site.xml\u3001mapred-site.xml\u3001yarn-site.xml\u3001workers\u6587\u4ef6，\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u4fee\u6539\u914d\u7f6e\u4fe1\u606f\u3002<\/p>\n
\u8be5\u6587\u4ef6\u4e2d\u8bbe\u7f6e\u7684\u662fHadoop\u8fd0\u884c\u65f6\u9700\u8981\u7684\u73af\u5883\u53d8\u91cf\u3002JAVA_HOME\u662f\u5fc5\u987b\u8bbe\u7f6e\u7684，\u5373\u4f7f\u6211\u4eec\u5f53\u524d\u7684\u7cfb\u7edf\u4e2d\u8bbe\u7f6e\u4e86JAVA_HOME ，\u5b83\u4e5f\u662f\u4e0d\u8ba4\u8bc6\u7684，\u56e0\u4e3aHadoop\u5373\u4f7f\u662f\u5728\u672c\u673a\u4e0a\u6267\u884c，\u5b83\u4e5f\u662f\u628a\u5f53\u524d\u7684\u6267\u884c\u73af\u5883\u5f53\u6210\u8fdc\u7a0b\u670d\u52a1\u5668\u3002<\/p>\n
vi $HADOOP_HOME\/etc\/hadoop\/hadoop-env.sh<\/p>\n
export JAVA_HOME=\/usr\/java\/jdk1.8.0_181
\nexport HDFS_NAMENODE_USER=hadoop
\nexport HDFS_DATANODE_USER=hadoop
\nexport HADOOP_SHELL_EXECNAME=hadoop
\nexport HDFS_SECONDARYNAMENODE_USER=hadoop
\nexport YARN_RESOURCEMANAGER_USER=hadoop
\nexport YARN_NODEMANAGER_USER=hadoop
\nexport HDFS_JOURNALNODE_USER=hadoop
\nexport HDFS_ZKFC_USER=hadoop<\/p>\n
\u8fd9\u4e00\u6b65\u53ef\u9009 vi $HADOOP_HOME\/etc\/hadoop\/yarn-env.sh<\/p>\n
export YARN_REGISTRYDNS_SECURE_USER=hadoop
\nexport YARN_RESOURCEMANAGER_USER=hadoop
\nexport YARN_NODEMANAGER_USER=hadoop<\/p>\n
vi $HADOOP_HOME\/etc\/hadoop\/core-site.xml<\/p>\n
<?xml version="1.0" encoding="UTF-8"?>
\n<?xml-stylesheet type="text\/xsl" href="configuration.xsl"?><\/p>\n
<!– Put site-specific property overrides in this file. –><\/p>\n
<configuration>
\n <!– \u6307\u5b9aNameNode\u7684\u5730\u5740 –>
\n <!– \u6307\u5b9anamenode\u7684hdfs\u534f\u8bae\u7684\u6587\u4ef6\u7cfb\u7edf\u901a\u4fe1\u5730\u5740 –>
\n <property>
\n <name>fs.defaultFS<\/name>
\n <value>hdfs:\/\/idc-bigdata-cluster<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6e hadoop.tmp.dir \u8def\u5f84\u5230\u6301\u4e45\u5316\u76ee\u5f55 –>
\n <!– \u5982\u679c\u4e0d\u6307\u5b9adfs.namenode.name.dir dfs.datanode.data.dir \u9ed8\u8ba4\u6570\u636e\u4f1a\u5b58hadoop.tmp.dir\u6240\u6307\u8def\u5f84\u4e0b\u7684dfs\u8def\u5f84\u4e2d –>
\n <property>
\n <name>hadoop.tmp.dir<\/name>
\n <value>\/datas\/dfs\/tmp<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6eHDFS\u7f51\u9875\u767b\u5f55\u4f7f\u7528\u7684\u9759\u6001\u7528\u6237\u4e3ahadoop –>
\n <property>
\n <name>hadoop.http.staticuser.user<\/name>
\n <value>hadoop<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6e\u8be5hadoop(superUser)\u5141\u8bb8\u901a\u8fc7\u4ee3\u7406\u8bbf\u95ee\u7684\u4e3b\u673a\u8282\u70b9 –>
\n <property>
\n <name>hadoop.proxyuser.hadoop.hosts<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6e\u8be5hadoop(superUser)\u5141\u8bb8\u901a\u8fc7\u4ee3\u7406\u7528\u6237\u6240\u5c5e\u7ec4 –>
\n <property>
\n <name>hadoop.proxyuser.hadoop.groups<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6e\u8be5hadoop(superUser)\u5141\u8bb8\u901a\u8fc7\u4ee3\u7406\u7684\u7528\u6237–>
\n <property>
\n <name>hadoop.proxyuser.hadoop.users<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u7f13\u51b2\u533a\u5927\u5c0f，\u5b9e\u9645\u5de5\u4f5c\u4e2d\u6839\u636e\u670d\u52a1\u5668\u6027\u80fd\u52a8\u6001\u8c03\u6574 –>
\n <property>
\n <name>io.file.buffer.size<\/name>
\n <value>131072<\/value>
\n <\/property><\/p>\n
<!– \u5f00\u542fhdfs\u7684\u5783\u573e\u6876\u673a\u5236，\u5220\u9664\u6389\u7684\u6570\u636e\u53ef\u4ee5\u4ece\u5783\u573e\u6876\u4e2d\u56de\u6536，\u5355\u4f4d\u5206\u949f –>
\n <property>
\n <name>fs.trash.interval<\/name>
\n <value>4320<\/value>
\n <\/property>
\n <property>
\n <name>fs.trash.checkpoint.interval<\/name>
\n <value>0<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9azkfc\u8981\u8fde\u63a5\u7684zkServer\u5730\u5740 –>
\n <property>
\n <name>ha.zookeeper.quorum<\/name>
\n <value>idc-bigdata-01:2181,idc-bigdata-02:2181,idc-bigdata-03:2181<\/value>
\n <\/property><\/p>\n
<\/configuration><\/p>\n
\u914d\u7f6e\u8bf4\u660e：<\/p>\n
fs.defaultFS \u6307\u5b9aHDFS\u4e2dNameNode\u7684\u5730\u5740
\nhadoop.tmp.dir \u6307\u5b9ahadoop\u8fd0\u884c\u65f6\u4ea7\u751f\u6587\u4ef6\u7684\u5b58\u50a8\u76ee\u5f55，\u662f\u5176\u4ed6\u4e34\u65f6\u76ee\u5f55\u7684\u7236\u76ee\u5f55 \u5982\u679c\u4e0d\u6307\u5b9adfs.namenode.name.dir dfs.datanode.data.dir \u9ed8\u8ba4\u6570\u636e\u4f1a\u5b58hadoop.tmp.dir\u6240\u6307\u8def\u5f84\u4e0b\u7684dfs\u8def\u5f84\u4e2d
\nha.zookeeper.quorum ZooKeeper\u5730\u5740\u5217\u8868，ZKFailoverController\u5c06\u5728\u81ea\u52a8\u6545\u969c\u8f6c\u79fb\u4e2d\u4f7f\u7528\u8fd9\u4e9b\u5730\u5740\u3002
\nio.file.buffer.size \u5728\u5e8f\u5217\u6587\u4ef6\u4e2d\u4f7f\u7528\u7684\u7f13\u51b2\u533a\u5927\u5c0f，\u6d41\u6587\u4ef6\u7684\u7f13\u51b2\u533a\u4e3a4K
\nfs.defaultFS：\u6307\u5b9anamenode\u7684hdfs\u534f\u8bae\u7684\u6587\u4ef6\u7cfb\u7edf\u901a\u4fe1\u5730\u5740，\u53ef\u4ee5\u6307\u5b9a\u4e00\u4e2a\u4e3b\u673a+\u7aef\u53e3
\nhadoop.tmp.dir：hadoop\u96c6\u7fa4\u5728\u5de5\u4f5c\u65f6\u5b58\u50a8\u7684\u4e00\u4e9b\u4e34\u65f6\u6587\u4ef6\u5b58\u653e\u7684\u76ee\u5f55<\/p>\n
vi $HADOOP_HOME\/etc\/hadoop\/hdfs-site.xml<\/p>\n
<?xml version="1.0" encoding="UTF-8"?>
\n<?xml-stylesheet type="text\/xsl" href="configuration.xsl"?><\/p>\n
<!– Put site-specific property overrides in this file. –><\/p>\n
<configuration><\/p>\n
<!– \u5b8c\u5168\u5206\u5e03\u5f0f\u96c6\u7fa4\u540d\u79f0 –>
\n <!–\u6307\u5b9ahdfs\u7684nameservice\u4e3aidc-bigdata-cluster，\u9700\u8981\u548ccore-site.xml\u4e2d\u7684\u4fdd\u6301\u4e00\u81f4 –>
\n <property>
\n <name>dfs.nameservices<\/name>
\n <value>idc-bigdata-cluster<\/value>
\n <\/property><\/p>\n
<!– \u96c6\u7fa4\u4e2dNameNode\u8282\u70b9\u90fd\u6709\u54ea\u4e9b –>
\n <!– namenode HA\u8282\u70b9\u522b\u540d –>
\n <property>
\n <name>dfs.ha.namenodes.idc-bigdata-cluster<\/name>
\n <value>nn1,nn2,nn3<\/value>
\n <\/property><\/p>\n
<!– NameNode\u7684RPC\u901a\u4fe1\u5730\u5740 –>
\n <!– \u5404\u4e2anamenode\u7684 rpc\u901a\u8baf\u5730\u5740 –>
\n <property>
\n <name>dfs.namenode.rpc-address.idc-bigdata-cluster.nn1<\/name>
\n <value>idc-bigdata-01:8020<\/value>
\n <\/property>
\n <property>
\n <name>dfs.namenode.rpc-address.idc-bigdata-cluster.nn2<\/name>
\n <value>idc-bigdata-02:8020<\/value>
\n <\/property>
\n <property>
\n <name>dfs.namenode.rpc-address.idc-bigdata-cluster.nn3<\/name>
\n <value>idc-bigdata-03:8020<\/value>
\n <\/property><\/p>\n
<!– NameNode\u7684http\u901a\u4fe1\u5730\u5740 –>
\n <!– \u5404\u4e2anamenode\u7684http\u72b6\u6001\u9875\u9762\u5730\u5740 –>
\n <property>
\n <name>dfs.namenode.http-address.idc-bigdata-cluster.nn1<\/name>
\n <value>idc-bigdata-01:9870<\/value>
\n <\/property>
\n <property>
\n <name>dfs.namenode.http-address.idc-bigdata-cluster.nn2<\/name>
\n <value>idc-bigdata-02:9870<\/value>
\n <\/property>
\n <property>
\n <name>dfs.namenode.http-address.idc-bigdata-cluster.nn3<\/name>
\n <value>idc-bigdata-03:9870<\/value>
\n <\/property><\/p>\n
<!– hdfs\u7684\u526f\u672c\u6570\u8bbe\u7f6e，\u9ed8\u8ba4\u4e3a3 –>
\n <property>
\n <name>dfs.replication<\/name>
\n <value>3<\/value>
\n <\/property><\/p>\n
<!– \u5927\u578b\u6587\u4ef6\u7cfb\u7edfHDFS\u5757\u5927\u5c0f\u4e3a256MB，\u9ed8\u8ba4\u662f128MB –>
\n <property>
\n <name>dfs.blocksize<\/name>
\n <value>134217728<\/value>
\n <\/property><\/p>\n
<!– NameNode\u6570\u636e\u5b58\u50a8\u76ee\u5f55 –>
\n <!– namenode\u6570\u636e\u7684\u5b58\u653e\u4f4d\u7f6e，\u5143\u6570\u636e\u5b58\u653e\u4f4d\u7f6e –>
\n <property>
\n <name>dfs.namenode.name.dir<\/name>
\n <value>file:\/\/\/datas\/dfs\/nn<\/value>
\n <\/property><\/p>\n
<!– DataNode\u6570\u636e\u5b58\u50a8\u76ee\u5f55 –>
\n <!– \u591a\u76ee\u5f55 file:\/\/\/data\/dfs\/dn1\/dn,file:\/\/\/data\/dfs\/dn2\/dn –>
\n <property>
\n <name>dfs.datanode.data.dir<\/name>
\n <value>\/datas\/dfs\/dn1,\/datas\/dfs\/dn2<\/value>
\n <\/property><\/p>\n
<!– JournalNode\u6570\u636e\u5b58\u50a8\u76ee\u5f55 –>
\n <property>
\n <name>dfs.journalnode.edits.dir<\/name>
\n <value>\/datas\/dfs\/jn<\/value>
\n <\/property><\/p>\n
<!– HA\u96c6\u7fa4\u4e2d\u591a\u4e2aNameNode\u4e4b\u95f4\u7684\u5171\u4eab\u5b58\u50a8\u4e0a\u7684\u76ee\u5f55 –>
\n <!– \u6307\u5b9aNameNode\u5143\u6570\u636e\u5728JournalNode\u4e0a\u7684\u5b58\u653e\u4f4d\u7f6e –>
\n <property>
\n <name>dfs.namenode.shared.edits.dir<\/name>
\n <value>qjournal:\/\/idc-bigdata-01:8485;idc-bigdata-02:8485;idc-bigdata-03:8485\/idc-bigdata-cluster<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6e\u5931\u8d25\u81ea\u52a8\u5207\u6362\u5b9e\u73b0\u65b9\u5f0f –>
\n <!– \u8bbf\u95ee\u4ee3\u7406\u7c7b：client\u7528\u4e8e\u786e\u5b9a\u54ea\u4e2aNameNode\u4e3aActive –>
\n <property>
\n <name>dfs.client.failover.proxy.provider.idc-bigdata-cluster<\/name>
\n <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<\/value>
\n <\/property><\/p>\n
<!– \u542f\u7528nn\u6545\u969c\u81ea\u52a8\u8f6c\u79fb –>
\n <!– \u5f00\u542fNameNode\u5931\u8d25\u81ea\u52a8\u5207\u6362 –>
\n <property>
\n <name>dfs.ha.automatic-failover.enabled<\/name>
\n <value>true<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6e\u9694\u79bb\u673a\u5236，\u5373\u540c\u4e00\u65f6\u523b\u53ea\u80fd\u6709\u4e00\u53f0\u670d\u52a1\u5668\u5bf9\u5916\u54cd\u5e94 –>
\n <property>
\n <name>dfs.ha.fencing.methods<\/name>
\n <value>sshfence
\n shell(\/bin\/true)
\n <\/value>
\n <\/property><\/p>\n
<!– \u4f7f\u7528\u9694\u79bb\u673a\u5236\u65f6\u9700\u8981ssh\u79d8\u94a5\u767b\u5f55–>
\n <!– \u4f7f\u7528sshfence\u9694\u79bb\u673a\u5236\u65f6\u9700\u8981ssh\u514d\u767b\u9646 –>
\n <property>
\n <name>dfs.ha.fencing.ssh.private-key-files<\/name>
\n <value>\/home\/hadoop\/.ssh\/id_rsa<\/value>
\n <\/property><\/p>\n
<property>
\n <name>dfs.ha.fencing.ssh.connect-timeout<\/name>
\n <value>30000<\/value>
\n <\/property><\/p>\n
<!–\u94fe\u63a5\u8fd4\u56dehostname\u7ed9\u5ba2\u6237\u7aef\u8fdb\u884c\u8fde\u63a5–>>
\n <property>
\n <name>dfs.client.use.datanode.hostname<\/name>
\n <value>true<\/value>
\n <\/property>
\n <!– \u5173\u95ed\u6743\u9650\u63a7\u5236 –>
\n <property>
\n <name>dfs.permissions<\/name>
\n <value>false<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9aDataNode\u7684\u8282\u70b9\u914d\u7f6e\u6587\u4ef6,\u9ed8\u8ba4\u4e3a\u6240\u6709 –>
\n <!– <property>
\n <name> dfs.hosts <\/name>
\n <value>\/opt\/module\/hadoop\/etc\/hadoop\/slaves<\/value>
\n <\/property> –>
\n<\/configuration><\/p>\n
\u914d\u7f6e\u8bf4\u660e:<\/p>\n
dfs.nameservices \u914d\u7f6e\u547d\u540d\u7a7a\u95f4，\u6240\u6709namenode\u8282\u70b9\u914d\u7f6e\u5728\u547d\u540d\u7a7a\u95f4mycluster\u4e0b
\ndfs.replication \u6307\u5b9adataNode\u5b58\u50a8block\u7684\u526f\u672c\u6570\u91cf，\u9ed8\u8ba4\u503c\u662f3\u4e2a
\ndfs.blocksize \u5927\u578b\u6587\u4ef6\u7cfb\u7edfHDFS\u5757\u5927\u5c0f\u4e3a256MB，\u9ed8\u8ba4\u662f128MB
\ndfs.namenode.rpc-address \u5404\u4e2anamenode\u7684 rpc\u901a\u8baf\u5730\u5740
\ndfs.namenode.http-address \u5404\u4e2anamenode\u7684http\u72b6\u6001\u9875\u9762\u5730\u5740
\ndfs.namenode.name.dir \u5b58\u653enamenode\u540d\u79f0\u8868（fsimage）\u7684\u76ee\u5f55
\ndfs.datanode.data.dir \u5b58\u653edatanode\u5757\u7684\u76ee\u5f55
\ndfs.namenode.shared.edits.dir HA\u96c6\u7fa4\u4e2d\u591a\u4e2aNameNode\u4e4b\u95f4\u7684\u5171\u4eab\u5b58\u50a8\u4e0a\u7684\u76ee\u5f55\u3002\u6b64\u76ee\u5f55\u5c06\u7531\u6d3b\u52a8\u670d\u52a1\u5668\u5199\u5165，\u7531\u5907\u7528\u670d\u52a1\u5668\u8bfb\u53d6，\u4ee5\u4fdd\u6301\u540d\u79f0\u7a7a\u95f4\u7684\u540c\u6b65\u3002
\ndfs.journalnode.edits.dir \u5b58\u50a8journal edit files\u7684\u76ee\u5f55
\ndfs.ha.automatic-failover.enabled \u662f\u5426\u542f\u7528\u6545\u969c\u81ea\u52a8\u5904\u7406
\ndfs.ha.fencing.methods \u5904\u4e8e\u6545\u969c\u72b6\u6001\u7684\u65f6\u5019hadoop\u8981\u9632\u6b62\u8111\u88c2\u95ee\u9898，\u6240\u4ee5\u5728standby\u673a\u5668\u5207\u6362\u5230active\u540e，hadoop\u8fd8\u4f1a\u8bd5\u56fe\u901a\u8fc7\u5185\u90e8\u7f51\u7edc\u7684ssh\u8fde\u8fc7\u53bb，\u5e76\u628anamenode\u7684\u76f8\u5173\u8fdb\u7a0b\u7ed9kill\u6389，\u4e00\u822c\u662fsshfence \u5c31\u662fssh\u65b9\u5f0f
\ndfs.ha.fencing.ssh.private-key-files \u914d\u7f6e\u4e86 ssh\u7528\u7684 key \u7684\u4f4d\u7f6e\u3002<\/p>\n
dfs.secondary.http.address：secondarynamenode\u8fd0\u884c\u8282\u70b9\u7684\u4fe1\u606f，\u5e94\u8be5\u548cnamenode\u5b58\u653e\u5728\u4e0d\u540c\u8282\u70b9
\ndfs.repliction：hdfs\u7684\u526f\u672c\u6570\u8bbe\u7f6e，\u9ed8\u8ba4\u4e3a3
\ndfs.namenode.name.dir：namenode\u6570\u636e\u7684\u5b58\u653e\u4f4d\u7f6e，\u5143\u6570\u636e\u5b58\u653e\u4f4d\u7f6e
\ndfs.datanode.data.dir：datanode\u6570\u636e\u7684\u5b58\u653e\u4f4d\u7f6e，block\u5757\u5b58\u653e\u7684\u4f4d\u7f6e<\/p>\n
vi $HADOOP_HOME\/etc\/hadoop\/mapred-site.xml<\/p>\n
<?xml version="1.0"?>
\n<?xml-stylesheet type="text\/xsl" href="configuration.xsl"?><\/p>\n
<!– Put site-specific property overrides in this file. –><\/p>\n
<configuration>
\n <!– \u6307\u5b9a\u5206\u5e03\u5f0f\u8ba1\u7b97\u4f7f\u7528\u7684\u6846\u67b6\u662fyarn –>
\n <property>
\n <name>mapreduce.framework.name<\/name>
\n <value>yarn<\/value>
\n <\/property><\/p>\n
<!– \u5f00\u542fMapReduce\u5c0f\u4efb\u52a1\u6a21\u5f0f –>
\n <property>
\n <name>mapreduce.job.ubertask.enable<\/name>
\n <value>true<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9a\u5386\u53f2\u670d\u52a1\u5668\u7684\u5730\u5740\u548c\u7aef\u53e3 –>
\n <property>
\n <name>mapreduce.jobhistory.address<\/name>
\n <value>idc-bigdata-03:10020<\/value>
\n <\/property><\/p>\n
<!– \u5386\u53f2\u670d\u52a1\u5668web\u7aef\u5730\u5740 –>
\n <!– \u67e5\u770b\u5386\u53f2\u670d\u52a1\u5668\u5df2\u7ecf\u8fd0\u884c\u5b8c\u7684Mapreduce\u4f5c\u4e1a\u8bb0\u5f55\u7684web\u5730\u5740 –>
\n <property>
\n <name>mapreduce.jobhistory.webapp.address<\/name>
\n <value>idc-bigdata-03:19888<\/value>
\n <\/property><\/p>\n
<property>
\n <name>mapreduce.map.memory.mb<\/name>
\n <value>1536<\/value>
\n <\/property>
\n <property>
\n <name>mapreduce.map.java.opts<\/name>
\n <value>-Xmx1024M<\/value>
\n <\/property>
\n <property>
\n\u3000\u3000 <name>mapreduce.reduce.memory.mb<\/name>
\n\u3000\u3000 <value>3072<\/value>
\n <\/property>
\n <property>
\n\u3000\u3000 <name>mapreduce.reduce.java.opts<\/name>
\n\u3000\u3000 <value>-Xmx2560M<\/value>
\n <\/property>
\n<\/configuration><\/p>\n
mapreduce.framework.name \u8bbe\u7f6eMapReduce\u8fd0\u884c\u5e73\u53f0\u4e3ayarn
\nmapreduce.jobhistory.address \u5386\u53f2\u670d\u52a1\u5668\u7684\u5730\u5740
\nmapreduce.jobhistory.webapp.address \u5386\u53f2\u670d\u52a1\u5668\u9875\u9762\u7684\u5730\u5740
\nmapreduce.framework.name：\u6307\u5b9amapreduce\u6846\u67b6\u4e3ayarn\u65b9\u5f0f
\nmapreduce.jobhistory.address：\u6307\u5b9a\u5386\u53f2\u670d\u52a1\u5668\u7684\u5730\u5740\u548c\u7aef\u53e3
\nmapreduce.jobhistory.webapp.address：\u67e5\u770b\u5386\u53f2\u670d\u52a1\u5668\u5df2\u7ecf\u8fd0\u884c\u5b8c\u7684Mapreduce\u4f5c\u4e1a\u8bb0\u5f55\u7684web\u5730\u5740，\u9700\u8981\u542f\u52a8\u8be5\u670d\u52a1\u624d\u884c
\n\u9ed8\u8ba4 MapReduce Job \u662f\u5b58\u5728\u5185\u5b58\u4e2d\u7684，\u91cd\u542f\u4e4b\u540e\u5c31\u4f1a\u4e22\u5931，\u6240\u4ee5\u9700\u8981\u914d\u7f6e\u5386\u53f2\u670d\u52a1\u5668，\u5c06\u4fe1\u606f\u5b58\u50a8\u5728 HDFS \u4e2d<\/p>\n
vi $HADOOP_HOME\/etc\/hadoop\/yarn-site.xml<\/p>\n
<?xml version="1.0"?>
\n<!–
\n Licensed under the Apache License, Version 2.0 (the "License");
\n you may not use this file except in compliance with the License.
\n You may obtain a copy of the License at<\/p>\n
http:\/\/www.apache.org\/licenses\/LICENSE-2.0<\/p>\n
Unless required by applicable law or agreed to in writing, software
\n distributed under the License is distributed on an "AS IS" BASIS,
\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
\n See the License for the specific language governing permissions and
\n limitations under the License. See accompanying LICENSE file.
\n–>
\n<configuration><\/p>\n
<!– Site specific YARN configuration properties –>
\n <!– \u6307\u5b9aMR\u8d70shuffle \u9ed8\u8ba4\u503cmapreduce_shuff –>
\n <property>
\n <name>yarn.nodemanager.aux-services<\/name>
\n <value>mapreduce_shuffle<\/value>
\n <\/property><\/p>\n
<!– rm\u5931\u8054\u540e\u91cd\u65b0\u94fe\u63a5\u7684\u65f6\u95f4 –>
\n <property>
\n <name>yarn.resourcemanager.connect.retry-interval.ms<\/name>
\n <value>2000<\/value>
\n <\/property><\/p>\n
<!– ========== \u542f\u7528\u96c6\u7fa4 ========== –>
\n <!– \u542f\u7528resourcemanager HA –>
\n <property>
\n <name>yarn.resourcemanager.ha.enabled<\/name>
\n <value>true<\/value>
\n <description>\u542f\u52a8Yran HA\u6a21\u5f0f<\/description>
\n <\/property><\/p>\n
<!– \u58f0\u660e\u4e24\u53f0resourcemanager\u7684\u5730\u5740 –>
\n <property>
\n <name>yarn.resourcemanager.cluster-id<\/name>
\n <value>idc-bigdata-yarn-cluster<\/value>
\n <\/property><\/p>\n
<!–\u6307\u5b9aresourcemanager\u7684\u903b\u8f91\u5217\u8868–>
\n <property>
\n <name>yarn.resourcemanager.ha.rm-ids<\/name>
\n <value>rm1,rm2<\/value>
\n <\/property><\/p>\n
<!– \u542f\u7528\u81ea\u52a8\u6062\u590d –>
\n <property>
\n <name>yarn.resourcemanager.recovery.enabled<\/name>
\n <value>true<\/value>
\n <\/property><\/p>\n
<!– \u5f00\u542fRM\u81ea\u52a8\u6545\u969c\u8f6c\u79fb –>
\n <property>
\n <name>yarn.resourcemanager.ha.automatic-failover.enabled<\/name>
\n <value>true<\/value>
\n <description>\u5f00\u542f ResourceManager \u6545\u969c\u81ea\u52a8\u5207\u6362<\/description>
\n <\/property><\/p>\n
<property>
\n <name>yarn.resourcemanager.ha.automatic-failover.embedded<\/name>
\n <value>true<\/value>
\n <\/property><\/p>\n
<!– ========== rm1\u7684\u914d\u7f6e ========== –>
\n <!– \u6307\u5b9arm1\u7684\u4e3b\u673a\u540d –>
\n <property>
\n <name>yarn.resourcemanager.hostname.rm1<\/name>
\n <value>idc-bigdata-01<\/value>
\n <\/property>
\n <!– \u6307\u5b9arm1\u7684web\u7aef\u5730\u5740 –>
\n <property>
\n <name>yarn.resourcemanager.webapp.address.rm1<\/name>
\n <value>idc-bigdata-01:8088<\/value>
\n <\/property>
\n <!– \u6307\u5b9arm1\u7684\u5185\u90e8\u901a\u4fe1\u5730\u5740 –>
\n <property>
\n <name>yarn.resourcemanager.address.rm1<\/name>
\n <value>idc-bigdata-01:8032<\/value>
\n <\/property>
\n <!– \u6307\u5b9aAM\u5411rm1\u7533\u8bf7\u8d44\u6e90\u7684\u5730\u5740 –>
\n <property>
\n <name>yarn.resourcemanager.scheduler.address.rm1<\/name>
\n <value>idc-bigdata-01:8030<\/value>
\n <\/property>
\n <!– \u6307\u5b9a\u4f9bNM\u8fde\u63a5\u7684\u5730\u5740 –>
\n <property>
\n <name>yarn.resourcemanager.resource-tracker.address.rm1<\/name>
\n <value>idc-bigdata-01:8031<\/value>
\n <\/property><\/p>\n
<!– ========== rm2\u7684\u914d\u7f6e ========== –>
\n <!– \u6307\u5b9arm2\u7684\u4e3b\u673a\u540d –>
\n <property>
\n <name>yarn.resourcemanager.hostname.rm2<\/name>
\n <value>idc-bigdata-02<\/value>
\n <\/property>
\n <property>
\n <name>yarn.resourcemanager.webapp.address.rm2<\/name>
\n <value>idc-bigdata-02:8088<\/value>
\n <\/property>
\n <property>
\n <name>yarn.resourcemanager.address.rm2<\/name>
\n <value>idc-bigdata-02:8032<\/value>
\n <\/property>
\n <property>
\n <name>yarn.resourcemanager.scheduler.address.rm2<\/name>
\n <value>idc-bigdata-02:8030<\/value>
\n <\/property>
\n <property>
\n <name>yarn.resourcemanager.resource-tracker.address.rm2<\/name>
\n <value>idc-bigdata-02:8031<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9azookeeper\u96c6\u7fa4\u7684\u5730\u5740 –>
\n <property>
\n <name>yarn.resourcemanager.zk-address<\/name>
\n <value>idc-bigdata-01:2181,idc-bigdata-02:2181,idc-bigdata-03:2181<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9aresourcemanager\u7684\u72b6\u6001\u4fe1\u606f\u5b58\u50a8\u5728zookeeper\u96c6\u7fa4 –>
\n <property>
\n <name>yarn.resourcemanager.store.class<\/name>
\n <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore<\/value>
\n <\/property><\/p>\n
<!– \u73af\u5883\u53d8\u91cf\u7684\u7ee7\u627f –>
\n <property>
\n <name>yarn.nodemanager.env-whitelist<\/name>
\n <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME<\/value>
\n <\/property><\/p>\n
<!– ResourceManager\u5904\u7406\u8c03\u5ea6\u5668\u8bf7\u6c42\u7684\u7ebf\u7a0b\u6570\u91cf,\u9ed8\u8ba450；\u5982\u679c\u63d0\u4ea4\u7684\u4efb\u52a1\u6570\u5927\u4e8e50，\u53ef\u4ee5\u589e\u52a0\u8be5\u503c，\u4f46\u662f\u4e0d\u80fd\u8d85\u8fc73\u53f0 * 4\u7ebf\u7a0b = 12\u7ebf\u7a0b（\u53bb\u9664\u5176\u4ed6\u5e94\u7528\u7a0b\u5e8f\u5b9e\u9645\u4e0d\u80fd\u8d85\u8fc78） –>
\n <property>
\n <description>Number of threads to handle scheduler interface.<\/description>
\n <name>yarn.resourcemanager.scheduler.client.thread-count<\/name>
\n <value>8<\/value>
\n <\/property><\/p>\n
<!–\u662f\u5426\u5c06\u865a\u62df\u6838\u6570\u5f53\u4f5cCPU\u6838\u6570，\u9ed8\u8ba4\u662ffalse，\u91c7\u7528\u7269\u7406CPU\u6838\u6570–>
\n <property>
\n <name>yarn.nodemanager.resource.count-logical-processors-as-cores<\/name>
\n <value>false<\/value>
\n <\/property><\/p>\n
<!– \u662f\u5426\u8ba9yarn\u81ea\u52a8\u68c0\u6d4b\u786c\u4ef6\u8fdb\u884c\u914d\u7f6e，\u9ed8\u8ba4\u662ffalse，\u5982\u679c\u8be5\u8282\u70b9\u6709\u5f88\u591a\u5176\u4ed6\u5e94\u7528\u7a0b\u5e8f，\u5efa\u8bae\u624b\u52a8\u914d\u7f6e\u3002\u5982\u679c\u8be5\u8282\u70b9\u6ca1\u6709\u5176\u4ed6\u5e94\u7528\u7a0b\u5e8f，\u53ef\u4ee5\u91c7\u7528\u81ea\u52a8 –>
\n <property>
\n <name>yarn.nodemanager.resource.detect-hardware-capabilities<\/name>
\n <value>false<\/value>
\n <\/property><\/p>\n
<!–Core\u8f6c\u6210Vcore\u7684\u4e2a\u6570（\u865a\u62df\u6838\u6570\u548c\u7269\u7406\u6838\u6570\u4e58\u6570，\u9ed8\u8ba4\u662f1.0）hadoop\u4e2d\u7684vcore\u4e0d\u662f\u771f\u6b63\u7684core，\u901a\u5e38vcore\u7684\u4e2a\u6570\u8bbe\u7f6e\u4e3a\u903b\u8f91cpu\u4e2a\u6570\u76841~5\u500d\u3002–>
\n <property>
\n <name>yarn.nodemanager.resource.pcores-vcores-multiplier<\/name>
\n <value>1.0<\/value>
\n <\/property><\/p>\n
<!– yarn\u5bb9\u5668\u5141\u8bb8\u5206\u914d\u7684\u6700\u5927\u6700\u5c0f\u5185\u5b58 –>
\n <!– \u5bb9\u5668\u6700\u5c0f\u5185\u5b58，\u9ed8\u8ba41G –>
\n <property>
\n <name>yarn.scheduler.minimum-allocation-mb<\/name>
\n <value>1024<\/value>
\n <\/property>
\n <!– yarn\u5bb9\u5668\u5141\u8bb8\u5206\u914d\u7684\u6700\u5927\u5185\u5b58，\u6839\u636e\u9700\u8981\u8c03\u6574 –>
\n <!– \u5bb9\u5668\u6700\u5927\u5185\u5b58，\u9ed8\u8ba48G，\u4fee\u6539\u4e3a10G –>
\n <property>
\n <name>yarn.scheduler.maximum-allocation-mb<\/name>
\n <value>10240<\/value>
\n <\/property><\/p>\n
<!– yarn\u5bb9\u5668\u5141\u8bb8\u7ba1\u7406\u7684\u7269\u7406\u5185\u5b58\u5927\u5c0f –>
\n <!– NodeManager\u4f7f\u7528\u5185\u5b58\u6570，\u9ed8\u8ba48G，\u4fee\u6539\u4e3a55G\u5185\u5b58 –>
\n <!– \u5355\u8282\u70b9\u53ef\u5206\u914d\u7684\u7269\u7406\u5185\u5b58\u603b\u91cf，\u9ed8\u8ba48G \u8fd9\u91cc\u662f64GB\u914d\u7f6e–>
\n <property>
\n <name>yarn.nodemanager.resource.memory-mb<\/name>
\n <value>56320<\/value>
\n <\/property><\/p>\n
<!– \u5bb9\u5668\u6700\u5c0fCPU\u6838\u6570，\u9ed8\u8ba41\u4e2a –>
\n <property>
\n <name>yarn.scheduler.minimum-allocation-vcores<\/name>
\n <value>1<\/value>
\n <\/property><\/p>\n
<!– nodemanager\u7684CPU\u6838\u6570，\u4e0d\u6309\u7167\u786c\u4ef6\u73af\u5883\u81ea\u52a8\u8bbe\u5b9a\u65f6\u9ed8\u8ba4\u662f8\u4e2a，\u4fee\u6539\u4e3a4\u4e2a –>
\n <!– \u5355\u8282\u70b9\u53ef\u5206\u914d\u7684\u865a\u62dfCPU\u4e2a\u6570，\u9ed8\u8ba48\u4e2a–>
\n <property>
\n <name>yarn.nodemanager.resource.cpu-vcores<\/name>
\n <value>4<\/value>
\n <\/property><\/p>\n
<!– \u5bb9\u5668\u6700\u5927CPU\u6838\u6570，\u9ed8\u8ba44\u4e2a –>
\n <property>
\n <name>yarn.scheduler.maximum-allocation-vcores<\/name>
\n <value>4<\/value>
\n <\/property><\/p>\n
<!– \u5173\u95edyarn\u5bf9\u7269\u7406\u5185\u5b58\u548c\u865a\u62df\u5185\u5b58\u7684\u9650\u5236\u68c0\u67e5 –>
\n <!– \u865a\u62df\u5185\u5b58\u68c0\u67e5(\u9650\u5236)，\u9ed8\u8ba4\u6253\u5f00，\u4fee\u6539\u4e3a\u5173\u95ed –>
\n <!–\u662f\u5426\u542f\u52a8\u4e00\u4e2a\u7ebf\u7a0b\u68c0\u67e5\u6bcf\u4e2a\u4efb\u52a1\u6b63\u4f7f\u7528\u7684\u7269\u7406\u5185\u5b58\u91cf，\u5982\u679c\u4efb\u52a1\u8d85\u51fa\u5206\u914d\u503c，\u5219\u76f4\u63a5\u5c06\u5176\u6740\u6389，\u9ed8\u8ba4\u662ftrue –>
\n <property>
\n <name>yarn.nodemanager.pmem-check-enabled<\/name>
\n <value>false<\/value>
\n <\/property>
\n <!–\u5173\u95edHadoop\u865a\u62df\u5185\u5b58\u68c0\u67e5 –>
\n <!–\u662f\u5426\u542f\u52a8\u4e00\u4e2a\u7ebf\u7a0b\u68c0\u67e5\u6bcf\u4e2a\u4efb\u52a1\u6b63\u4f7f\u7528\u7684\u865a\u62df\u5185\u5b58\u91cf，\u5982\u679c\u4efb\u52a1\u8d85\u51fa\u5206\u914d\u503c，\u5219\u76f4\u63a5\u5c06\u5176\u6740\u6389，\u9ed8\u8ba4\u662ftrue –>
\n <property>
\n <name>yarn.nodemanager.vmem-check-enabled<\/name>
\n <value>false<\/value>
\n <\/property>
\n <!– \u865a\u62df\u5185\u5b58\u548c\u7269\u7406\u5185\u5b58\u8bbe\u7f6e\u6bd4\u4f8b,\u9ed8\u8ba42.1 –>
\n <property>
\n <name>yarn.nodemanager.vmem-pmem-ratio<\/name>
\n <value>2.1<\/value>
\n <\/property><\/p>\n
<!– \u5f00\u542f\u65e5\u5fd7\u805a\u96c6\u529f\u80fd –>
\n <property>
\n <name>yarn.log-aggregation-enable<\/name>
\n <value>true<\/value>
\n <\/property><\/p>\n
<!– \u8bbe\u7f6e\u65e5\u5fd7\u805a\u96c6\u670d\u52a1\u5668\u5730\u5740 –>
\n <!–\u5141\u8bb8\u7b2c\u4e09\u65b9\u7a0b\u5e8f，\u4f8b\u5982spark\u5c06Job\u7684\u65e5\u5fd7，\u63d0\u4ea4\u7ed9Hadoop\u7684\u5386\u53f2\u670d\u52a1 –>
\n <property>
\n <name>yarn.log.server.url<\/name>
\n <value>http:\/\/idc-bigdata-03:19888\/jobhistory\/logs<\/value>
\n <\/property><\/p>\n
<!– \u8bbe\u7f6e\u65e5\u5fd7\u4fdd\u7559\u65f6\u95f4\u4e3a7\u5929 –>
\n <!– \u914d\u7f6e\u65e5\u5fd7\u8fc7\u671f\u65f6\u95f4,\u5355\u4f4d\u79d2 –>
\n <property>
\n <name>yarn.log-aggregation.retain-seconds<\/name>
\n <value>604800<\/value>
\n <\/property><\/p>\n
<property>
\n <name>yarn.acl.enable<\/name>
\n <value>false<\/value>
\n <\/property><\/p>\n
<\/configuration><\/p>\n
\u521b\u5efa\u4e24\u4e2a\u961f\u5217\u5206\u522b\u4e3aonline\u548coffline，\u5c06\u8fd9\u4e24\u4e2a\u961f\u5217\u7684\u8d44\u6e90\u5206\u522b\u5206\u914d\u4e3a40%\u300140%，\u4e14\u5141\u8bb8\u5728\u8d44\u6e90\u4e0d\u8db3\u65f6\u501f\u7528\u5176\u4ed6\u961f\u5217\u7684\u8d44\u6e90， online\u961f\u5217\u91cc\u9762\u8fd0\u884c\u5b9e\u65f6\u4efb\u52a1 offline\u961f\u5217\u91cc\u9762\u8fd0\u884c\u79bb\u7ebf\u4efb\u52a1<\/p>\n
vi $HADOOP_HOME\/etc\/hadoop\/capacity-scheduler.xml<\/p>\n
<configuration><\/p>\n
<!– \u8bbe\u7f6e\u7cfb\u7edf\u4e2d\u53ef\u4ee5\u540c\u65f6\u8fd0\u884c\u548c\u7b49\u5f85\u7684\u5e94\u7528\u6570\u91cf\u3002\u9ed8\u8ba4\u662f10000 –>
\n <property>
\n <name>yarn.scheduler.capacity.maximum-applications<\/name>
\n <value>10000<\/value>
\n <\/property><\/p>\n
<!– \u8868\u793a\u96c6\u7fa4\u4e0a\u67d0\u961f\u5217\u53ef\u4f7f\u7528\u7684\u8d44\u6e90\u6bd4\u4f8b \u76ee\u7684\u662f\u4e3a\u4e86\u9650\u5236\u8fc7\u591a\u7684am\u6570，\u5373app\u6570 –>
\n <property>
\n <name>yarn.scheduler.capacity.maximum-am-resource-percent<\/name>
\n <value>0.2<\/value>
\n <\/property><\/p>\n
<!– \u914d\u7f6e\u6307\u5b9a\u8c03\u5ea6\u5668\u4f7f\u7528\u7684\u8d44\u6e90\u8ba1\u7b97\u5668 –>
\n <!– DefaultResourseCalculator \u9ed8\u8ba4\u503c,\u53ea\u4f7f\u7528\u5185\u5b58\u8fdb\u884c\u6bd4\u8f83 –>
\n <!– DominantResourceCalculator \u591a\u7ef4\u5ea6\u8d44\u6e90\u8ba1\u7b97,\u5185\u5b58\u3001cpu –>
\n <property>
\n <name>yarn.scheduler.capacity.resource-calculator<\/name>
\n <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9a\u591a\u4e2a\u961f\u5217，\u589e\u52a0hive,spark,flink\u961f\u5217 –>
\n <!– root\u961f\u5217\u4e2d\u6709\u54ea\u4e9b\u5b50\u961f\u5217–>
\n <property>
\n <name>yarn.scheduler.capacity.root.queues<\/name>
\n <value>default,online,offline<\/value>
\n <\/property><\/p>\n
<!– ******************************default\u961f\u5217**************************************** –><\/p>\n
<!– \u8bbe\u7f6edefault\u961f\u5217\u5360\u603b\u5185\u5b58\u768410%，\u9ed8\u8ba4100% \u603b\u548c\u4e3a100% –>
\n <property>
\n <name>yarn.scheduler.capacity.root.default.capacity<\/name>
\n <value>20<\/value>
\n <\/property><\/p>\n
<!– \u8bbe\u7f6e\u961f\u5217\u5f39\u6027\u8c03\u5ea6\u8d44\u6e90\u6700\u5927\u5bb9\u91cf –>
\n <!– deault\u961f\u5217\u7684\u6700\u5927\u5bb9\u91cf;\u81ea\u5df1\u961f\u5217\u8d44\u6e90\u4e0d\u591f，\u53ef\u4ee5\u4f7f\u7528\u5176\u4ed6\u961f\u5217\u8d44\u6e90\u4e0a\u9650\u767e\u5206\u6bd480% –><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.root.default.maximum-capacity<\/name>
\n <value>80<\/value>
\n <\/property><\/p>\n
<!– \u7528\u6237\u6700\u591a\u80fd\u591f\u83b7\u53d6\u8be5\u961f\u5217\u8d44\u6e90\u5bb9\u91cf\u7684\u6bd4\u4f8b，\u53d6\u503c0-1 –>
\n <!– \u5141\u8bb8\u5355\u4e2a\u7528\u6237\u6700\u591a\u53ef\u83b7\u53d6\u7684\u961f\u5217\u8d44\u6e90\u7684\u500d\u6570，\u9ed8\u8ba4\u503c1，\u786e\u4fdd\u5355\u4e2a\u7528\u6237\u65e0\u8bba\u96c6\u7fa4\u6709\u591a\u7a7a\u95f2，\u6c38\u8fdc\u4e0d\u4f1a\u5360\u7528\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u8d44\u6e90\u5f53\u503c\u5927\u4e8e1\u65f6，\u7528\u6237\u53ef\u4f7f\u7528\u7684\u8d44\u6e90\u5c06\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u8d44\u6e90
\n，\u4f46\u5e94\u8be5\u4e0d\u80fd\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u6700\u5927\u8d44\u6e90–>
\n <property>
\n <name>yarn.scheduler.capacity.root.default.user-limit-factor<\/name>
\n <value>1<\/value>
\n <\/property><\/p>\n
<!– \u542f\u52a8hive\u961f\u5217 \u961f\u5217\u72b6\u6001：RUNNING \u505c\u7528\u72b6\u6001：STOPPED –>
\n <property>
\n <name>yarn.scheduler.capacity.root.default.state<\/name>
\n <value>RUNNING<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u5411\u961f\u5217\u63d0\u4ea4\u4f5c\u4e1a –>
\n <!– \u8bbf\u95ee\u63a7\u5236，\u63a7\u5236\u8c01\u53ef\u4ee5\u5c06\u4efb\u52a1\u63d0\u4ea4\u5230\u8be5\u961f\u5217,*\u8868\u793a\u4efb\u4f55\u4eba –>
\n <property>
\n <name>yarn.scheduler.capacity.root.default.acl_submit_applications<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u64cd\u4f5c\u961f\u5217，\u7ba1\u7406\u5458\u6743\u9650（\u67e5\u770b\/\u6740\u6b7b）–>
\n <!– \u8bbf\u95ee\u63a7\u5236，\u63a7\u5236\u8c01\u53ef\u4ee5\u7ba1\u7406(\u5305\u62ec\u63d0\u4ea4\u548c\u53d6\u6d88)\u8be5\u961f\u5217\u7684\u4efb\u52a1，*\u8868\u793a\u4efb\u4f55\u4eba –>
\n <!– \u4e3aroot\u961f\u5217\u6307\u5b9a\u4e00\u4e2a\u7ba1\u7406\u5458，\u8be5\u7ba1\u7406\u5458\u53ef\u63a7\u5236\u8be5\u961f\u5217\u7684\u6240\u6709\u5e94\u7528\u7a0b\u5e8f，\u6bd4\u5982\u6740\u6b7b\u4efb\u610f\u4e00\u4e2a\u5e94\u7528\u7a0b\u5e8f\u7b49 –><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.root.default.acl_administer_queue<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u914d\u7f6e\u63d0\u4ea4\u4efb\u52a1\u4f18\u5148\u7ea7 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.default.acl_application_max_priority<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9a\u4efb\u52a1\u8fd0\u884c\u7684\u6700\u5927\u65f6\u95f4 –>
\n <!– \u4efb\u52a1\u7684\u8d85\u65f6\u65f6\u95f4\u8bbe\u7f6e：yarn application -appId ${appId} -updateLifeTime Timeout –>
\n <!– \u5982\u679capplication\u6307\u5b9a\u4e86\u8d85\u65f6\u65f6\u95f4，\u5219\u63d0\u4ea4\u5230\u8be5\u961f\u5217\u7684application\u80fd\u591f\u5236\u5b9a\u7684\u6700\u5927\u8d85\u65f6\u65f6\u95f4\u4e0d\u80fd\u8d85\u8fc7\u8be5\u503c\u3002–>
\n <!– default\u961f\u5217\u4e2d\u4efb\u52a1\u7684\u6700\u5927\u751f\u547d\u65f6\u957f，\u4ee5\u79d2\u4e3a\u5355\u4f4d\u3002\u4efb\u4f55\u5c0f\u4e8e\u6216\u7b49\u4e8e\u96f6\u7684\u503c\u5c06\u88ab\u89c6\u4e3a\u7981\u7528 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.default.maximum-application-lifetime<\/name>
\n <value>-1<\/value>
\n <\/property><\/p>\n
<!– \u5982\u679capplication\u6ca1\u6709\u6307\u5b9a\u8d85\u65f6\u65f6\u95f4，\u5219\u7528default-application-lifetime \u4f5c\u4e3a\u9ed8\u8ba4\u503c –>
\n <!– hive\u961f\u5217\u4e2d\u4efb\u52a1\u7684\u9ed8\u8ba4\u751f\u547d\u65f6\u957f，\u4ee5\u79d2\u4e3a\u5355\u4f4d\u3002\u4efb\u4f55\u5c0f\u4e8e\u6216\u7b49\u4e8e\u96f6\u7684\u503c\u5c06\u88ab\u89c6\u4e3a\u7981\u7528 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.default.default-application-lifetime<\/name>
\n <value>-1<\/value>
\n <\/property><\/p>\n
<!– ******************************online\u961f\u5217**************************************** –><\/p>\n
<!– \u8bbe\u7f6eonline\u961f\u5217\u5360\u603b\u5185\u5b58\u768410%，\u9ed8\u8ba4100% \u603b\u548c\u4e3a100% –>
\n <property>
\n <name>yarn.scheduler.capacity.root.online.capacity<\/name>
\n <value>40<\/value>
\n <\/property><\/p>\n
<!– \u8bbe\u7f6e\u961f\u5217\u5f39\u6027\u8c03\u5ea6\u8d44\u6e90\u6700\u5927\u5bb9\u91cf –>
\n <!– deault\u961f\u5217\u7684\u6700\u5927\u5bb9\u91cf;\u81ea\u5df1\u961f\u5217\u8d44\u6e90\u4e0d\u591f，\u53ef\u4ee5\u4f7f\u7528\u5176\u4ed6\u961f\u5217\u8d44\u6e90\u4e0a\u9650\u767e\u5206\u6bd480% –><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.root.online.maximum-capacity<\/name>
\n <value>80<\/value>
\n <\/property><\/p>\n
<!– \u7528\u6237\u6700\u591a\u80fd\u591f\u83b7\u53d6\u8be5\u961f\u5217\u8d44\u6e90\u5bb9\u91cf\u7684\u6bd4\u4f8b，\u53d6\u503c0-1 –>
\n <!– \u5141\u8bb8\u5355\u4e2a\u7528\u6237\u6700\u591a\u53ef\u83b7\u53d6\u7684\u961f\u5217\u8d44\u6e90\u7684\u500d\u6570，\u9ed8\u8ba4\u503c1，\u786e\u4fdd\u5355\u4e2a\u7528\u6237\u65e0\u8bba\u96c6\u7fa4\u6709\u591a\u7a7a\u95f2，\u6c38\u8fdc\u4e0d\u4f1a\u5360\u7528\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u8d44\u6e90\u5f53\u503c\u5927\u4e8e1\u65f6，\u7528\u6237\u53ef\u4f7f\u7528\u7684\u8d44\u6e90\u5c06\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u8d44\u6e90
\n，\u4f46\u5e94\u8be5\u4e0d\u80fd\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u6700\u5927\u8d44\u6e90–>
\n <property>
\n <name>yarn.scheduler.capacity.root.online.user-limit-factor<\/name>
\n <value>1<\/value>
\n <\/property><\/p>\n
<!– \u542f\u52a8hive\u961f\u5217 \u961f\u5217\u72b6\u6001：RUNNING \u505c\u7528\u72b6\u6001：STOPPED –>
\n <property>
\n <name>yarn.scheduler.capacity.root.online.state<\/name>
\n <value>RUNNING<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u5411\u961f\u5217\u63d0\u4ea4\u4f5c\u4e1a –>
\n <!– \u8bbf\u95ee\u63a7\u5236，\u63a7\u5236\u8c01\u53ef\u4ee5\u5c06\u4efb\u52a1\u63d0\u4ea4\u5230\u8be5\u961f\u5217,*\u8868\u793a\u4efb\u4f55\u4eba –>
\n <property>
\n <name>yarn.scheduler.capacity.root.online.acl_submit_applications<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u64cd\u4f5c\u961f\u5217，\u7ba1\u7406\u5458\u6743\u9650（\u67e5\u770b\/\u6740\u6b7b）–>
\n <!– \u8bbf\u95ee\u63a7\u5236，\u63a7\u5236\u8c01\u53ef\u4ee5\u7ba1\u7406(\u5305\u62ec\u63d0\u4ea4\u548c\u53d6\u6d88)\u8be5\u961f\u5217\u7684\u4efb\u52a1，*\u8868\u793a\u4efb\u4f55\u4eba –>
\n <!– \u4e3aroot\u961f\u5217\u6307\u5b9a\u4e00\u4e2a\u7ba1\u7406\u5458，\u8be5\u7ba1\u7406\u5458\u53ef\u63a7\u5236\u8be5\u961f\u5217\u7684\u6240\u6709\u5e94\u7528\u7a0b\u5e8f，\u6bd4\u5982\u6740\u6b7b\u4efb\u610f\u4e00\u4e2a\u5e94\u7528\u7a0b\u5e8f\u7b49 –><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.root.online.acl_administer_queue<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u914d\u7f6e\u63d0\u4ea4\u4efb\u52a1\u4f18\u5148\u7ea7 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.online.acl_application_max_priority<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9a\u4efb\u52a1\u8fd0\u884c\u7684\u6700\u5927\u65f6\u95f4 –>
\n <!– \u4efb\u52a1\u7684\u8d85\u65f6\u65f6\u95f4\u8bbe\u7f6e：yarn application -appId ${appId} -updateLifeTime Timeout –>
\n <!– \u5982\u679capplication\u6307\u5b9a\u4e86\u8d85\u65f6\u65f6\u95f4，\u5219\u63d0\u4ea4\u5230\u8be5\u961f\u5217\u7684application\u80fd\u591f\u5236\u5b9a\u7684\u6700\u5927\u8d85\u65f6\u65f6\u95f4\u4e0d\u80fd\u8d85\u8fc7\u8be5\u503c\u3002–>
\n <!– default\u961f\u5217\u4e2d\u4efb\u52a1\u7684\u6700\u5927\u751f\u547d\u65f6\u957f，\u4ee5\u79d2\u4e3a\u5355\u4f4d\u3002\u4efb\u4f55\u5c0f\u4e8e\u6216\u7b49\u4e8e\u96f6\u7684\u503c\u5c06\u88ab\u89c6\u4e3a\u7981\u7528 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.online.maximum-application-lifetime<\/name>
\n <value>-1<\/value>
\n <\/property><\/p>\n
<!– \u5982\u679capplication\u6ca1\u6709\u6307\u5b9a\u8d85\u65f6\u65f6\u95f4，\u5219\u7528default-application-lifetime \u4f5c\u4e3a\u9ed8\u8ba4\u503c –>
\n <!– hive\u961f\u5217\u4e2d\u4efb\u52a1\u7684\u9ed8\u8ba4\u751f\u547d\u65f6\u957f，\u4ee5\u79d2\u4e3a\u5355\u4f4d\u3002\u4efb\u4f55\u5c0f\u4e8e\u6216\u7b49\u4e8e\u96f6\u7684\u503c\u5c06\u88ab\u89c6\u4e3a\u7981\u7528 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.online.default-application-lifetime<\/name>
\n <value>-1<\/value>
\n <\/property><\/p>\n
<!– ******************************offline\u961f\u5217**************************************** –><\/p>\n
<!– \u8bbe\u7f6eoffline\u961f\u5217\u5360\u603b\u5185\u5b58\u768440%，\u9ed8\u8ba4100% \u603b\u548c\u4e3a100% –>
\n <property>
\n <name>yarn.scheduler.capacity.root.offline.capacity<\/name>
\n <value>40<\/value>
\n <\/property><\/p>\n
<!– \u8bbe\u7f6e\u961f\u5217\u5f39\u6027\u8c03\u5ea6\u8d44\u6e90\u6700\u5927\u5bb9\u91cf –>
\n <!– deault\u961f\u5217\u7684\u6700\u5927\u5bb9\u91cf;\u81ea\u5df1\u961f\u5217\u8d44\u6e90\u4e0d\u591f，\u53ef\u4ee5\u4f7f\u7528\u5176\u4ed6\u961f\u5217\u8d44\u6e90\u4e0a\u9650\u767e\u5206\u6bd480% –><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.root.offline.maximum-capacity<\/name>
\n <value>80<\/value>
\n <\/property><\/p>\n
<!– \u7528\u6237\u6700\u591a\u80fd\u591f\u83b7\u53d6\u8be5\u961f\u5217\u8d44\u6e90\u5bb9\u91cf\u7684\u6bd4\u4f8b，\u53d6\u503c0-1 –>
\n <!– \u5141\u8bb8\u5355\u4e2a\u7528\u6237\u6700\u591a\u53ef\u83b7\u53d6\u7684\u961f\u5217\u8d44\u6e90\u7684\u500d\u6570，\u9ed8\u8ba4\u503c1，\u786e\u4fdd\u5355\u4e2a\u7528\u6237\u65e0\u8bba\u96c6\u7fa4\u6709\u591a\u7a7a\u95f2，\u6c38\u8fdc\u4e0d\u4f1a\u5360\u7528\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u8d44\u6e90\u5f53\u503c\u5927\u4e8e1\u65f6，\u7528\u6237\u53ef\u4f7f\u7528\u7684\u8d44\u6e90\u5c06\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u8d44\u6e90
\n，\u4f46\u5e94\u8be5\u4e0d\u80fd\u8d85\u8fc7\u961f\u5217\u914d\u7f6e\u7684\u6700\u5927\u8d44\u6e90–>
\n <property>
\n <name>yarn.scheduler.capacity.root.offline.user-limit-factor<\/name>
\n <value>1<\/value>
\n <\/property><\/p>\n
<!– \u542f\u52a8hive\u961f\u5217 \u961f\u5217\u72b6\u6001：RUNNING \u505c\u7528\u72b6\u6001：STOPPED –>
\n <property>
\n <name>yarn.scheduler.capacity.root.offline.state<\/name>
\n <value>RUNNING<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u5411\u961f\u5217\u63d0\u4ea4\u4f5c\u4e1a –>
\n <!– \u8bbf\u95ee\u63a7\u5236，\u63a7\u5236\u8c01\u53ef\u4ee5\u5c06\u4efb\u52a1\u63d0\u4ea4\u5230\u8be5\u961f\u5217,*\u8868\u793a\u4efb\u4f55\u4eba –>
\n <property>
\n <name>yarn.scheduler.capacity.root.offline.acl_submit_applications<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u64cd\u4f5c\u961f\u5217，\u7ba1\u7406\u5458\u6743\u9650（\u67e5\u770b\/\u6740\u6b7b）–>
\n <!– \u8bbf\u95ee\u63a7\u5236，\u63a7\u5236\u8c01\u53ef\u4ee5\u7ba1\u7406(\u5305\u62ec\u63d0\u4ea4\u548c\u53d6\u6d88)\u8be5\u961f\u5217\u7684\u4efb\u52a1，*\u8868\u793a\u4efb\u4f55\u4eba –>
\n <!– \u4e3aroot\u961f\u5217\u6307\u5b9a\u4e00\u4e2a\u7ba1\u7406\u5458，\u8be5\u7ba1\u7406\u5458\u53ef\u63a7\u5236\u8be5\u961f\u5217\u7684\u6240\u6709\u5e94\u7528\u7a0b\u5e8f，\u6bd4\u5982\u6740\u6b7b\u4efb\u610f\u4e00\u4e2a\u5e94\u7528\u7a0b\u5e8f\u7b49 –><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.root.offline.acl_administer_queue<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u54ea\u4e9b\u7528\u6237\u6709\u6743\u914d\u7f6e\u63d0\u4ea4\u4efb\u52a1\u4f18\u5148\u7ea7 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.offline.acl_application_max_priority<\/name>
\n <value>*<\/value>
\n <\/property><\/p>\n
<!– \u6307\u5b9a\u4efb\u52a1\u8fd0\u884c\u7684\u6700\u5927\u65f6\u95f4 –>
\n <!– \u4efb\u52a1\u7684\u8d85\u65f6\u65f6\u95f4\u8bbe\u7f6e：yarn application -appId ${appId} -updateLifeTime Timeout –>
\n <!– \u5982\u679capplication\u6307\u5b9a\u4e86\u8d85\u65f6\u65f6\u95f4，\u5219\u63d0\u4ea4\u5230\u8be5\u961f\u5217\u7684application\u80fd\u591f\u5236\u5b9a\u7684\u6700\u5927\u8d85\u65f6\u65f6\u95f4\u4e0d\u80fd\u8d85\u8fc7\u8be5\u503c\u3002–>
\n <!– default\u961f\u5217\u4e2d\u4efb\u52a1\u7684\u6700\u5927\u751f\u547d\u65f6\u957f，\u4ee5\u79d2\u4e3a\u5355\u4f4d\u3002\u4efb\u4f55\u5c0f\u4e8e\u6216\u7b49\u4e8e\u96f6\u7684\u503c\u5c06\u88ab\u89c6\u4e3a\u7981\u7528 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.offline.maximum-application-lifetime<\/name>
\n <value>-1<\/value>
\n <\/property><\/p>\n
<!– \u5982\u679capplication\u6ca1\u6709\u6307\u5b9a\u8d85\u65f6\u65f6\u95f4，\u5219\u7528default-application-lifetime \u4f5c\u4e3a\u9ed8\u8ba4\u503c –>
\n <!– hive\u961f\u5217\u4e2d\u4efb\u52a1\u7684\u9ed8\u8ba4\u751f\u547d\u65f6\u957f，\u4ee5\u79d2\u4e3a\u5355\u4f4d\u3002\u4efb\u4f55\u5c0f\u4e8e\u6216\u7b49\u4e8e\u96f6\u7684\u503c\u5c06\u88ab\u89c6\u4e3a\u7981\u7528 –>
\n <property>
\n <name>yarn.scheduler.capacity.root.offline.default-application-lifetime<\/name>
\n <value>-1<\/value>
\n <\/property><\/p>\n
<!– ============================= \u5176\u5b83\u914d\u7f6e ==================================== –><\/p>\n
<!–CapacityScheduler\u5c1d\u8bd5\u8c03\u5ea6\u673a\u672c\u5730\u5bb9\u5668\u4e4b\u540e\u9519\u8fc7\u7684\u8c03\u5ea6\u673a\u4f1a\u6570\u3002\u901a\u5e38，\u5e94\u8be5\u5c06\u5176\u8bbe\u7f6e\u4e3a\u96c6\u7fa4\u4e2d\u7684\u8282\u70b9\u6570\u3002\u9ed8\u8ba4\u60c5\u51b5\u4e0b\u5728\u4e00\u4e2a\u67b6\u6784\u4e2d\u8bbe\u7f6e\u5927\u7ea640\u4e2a\u8282\u70b9\u3002\u5e94\u4e3a\u6b63\u6574\u6570\u503c\u3002–>
\n <property>
\n <name>yarn.scheduler.capacity.node-locality-delay<\/name>
\n <value>40<\/value>
\n <\/property><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.rack-locality-additional-delay<\/name>
\n <value>-1<\/value>
\n <description>
\n Number of additional missed scheduling opportunities over the node-locality-delay
\n ones, after which the CapacityScheduler attempts to schedule off-switch containers,
\n instead of rack-local ones.
\n Example: with node-locality-delay=40 and rack-locality-delay=20, the scheduler will
\n attempt rack-local assignments after 40 missed opportunities, and off-switch assignments
\n after 40+20=60 missed opportunities.
\n When setting this parameter, the size of the cluster should be taken into account.
\n We use -1 as the default value, which disables this feature. In this case, the number
\n of missed opportunities for assigning off-switch containers is calculated based on
\n the number of containers and unique locations specified in the resource request,
\n as well as the size of the cluster.
\n <\/description>
\n <\/property><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.queue-mappings<\/name>
\n <value><\/value>
\n <description>
\n A list of mappings that will be used to assign jobs to queues
\n The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
\n Typically this list will be used to map users to queues,
\n for example, u:%user:%user maps all users to queues with the same name
\n as the user.
\n <\/description>
\n <\/property><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.queue-mappings-override.enable<\/name>
\n <value>false<\/value>
\n <description>
\n If a queue mapping is present, will it override the value specified
\n by the user? This can be used by administrators to place jobs in queues
\n that are different than the one specified by the user.
\n The default is false.
\n <\/description>
\n <\/property><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments<\/name>
\n <value>1<\/value>
\n <description>
\n Controls the number of OFF_SWITCH assignments allowed
\n during a node's heartbeat. Increasing this value can improve
\n scheduling rate for OFF_SWITCH containers. Lower values reduce
\n "clumping" of applications on particular nodes. The default is 1.
\n Legal values are 1-MAX_INT. This config is refreshable.
\n <\/description>
\n <\/property><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.application.fail-fast<\/name>
\n <value>false<\/value>
\n <description>
\n Whether RM should fail during recovery if previous applications'
\n queue is no longer valid.
\n <\/description>
\n <\/property><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.workflow-priority-mappings<\/name>
\n <value><\/value>
\n <description>
\n A list of mappings that will be used to override application priority.
\n The syntax for this list is
\n [workflowId]:[full_queue_name]:[priority][,next mapping]*
\n where an application submitted (or mapped to) queue "full_queue_name"
\n and workflowId "workflowId" (as specified in application submission
\n context) will be given priority "priority".
\n <\/description>
\n <\/property><\/p>\n
<property>
\n <name>yarn.scheduler.capacity.workflow-priority-mappings-override.enable<\/name>
\n <value>false<\/value>
\n <description>
\n If a priority mapping is present, will it override the value specified
\n by the user? This can be used by administrators to give applications a
\n priority that is different than the one specified by the user.
\n The default is false.
\n <\/description>
\n <\/property><\/p>\n
<\/configuration><\/p>\n
cat > $HADOOP_HOME\/etc\/hadoop\/workers <<EOF
\nidc-bigdata-01
\nidc-bigdata-02
\nidc-bigdata-03
\nEOF<\/p>\n
\u914d\u7f6e\u8bf4\u660e workers \u914d\u7f6edatanode\u5de5\u4f5c\u7684\u673a\u5668，\u800cdatanode\u4e3b\u8981\u662f\u7528\u6765\u5b58\u653e\u6570\u636e\u6587\u4ef6\u7684<\/p>\n
for host in idc-bigdata-{02..03}
\ndo
\necho =========$host hadoop install=======================
\nscp -r \/opt\/module\/hadoop-3.3.5 $host:\/opt\/module
\nssh $host ln -sf \/opt\/module\/hadoop-3.3.5 \/opt\/module\/hadoop
\nssh $host chown -R hadoop:hadoop \/opt\/module\/hadoop*
\nscp -r \/etc\/profile.d\/myenv.sh $host:\/etc\/profile.d\/myenv.sh
\nssh $host source \/etc\/profile.d\/myenv.sh
\necho
\ndone<\/p>\n
parted \/dev\/sdb
\nmklabel gpt
\nmkpart primary xfs 0% 100%
\nmkfs.xfs \/dev\/sdb1<\/p>\n
parted \/dev\/sdc
\nmklabel gpt
\nmkpart primary xfs 0% 100%
\nmkfs.xfs \/dev\/sdc1<\/p>\n
mkdir -p \/datas\/dfs\/{dn1,dn2}
\nchown -R hadoop:hadoop \/datas\/dfs<\/p>\n
\u6dfb\u52a0\u5f00\u673a\u6302\u8f7d<\/p>\n
vim \/etc\/fstab <\/p>\n
\/dev\/sdb1 \/datas\/dfs\/dn1 xfs defaults 0 0
\n\/dev\/sdc1 \/datas\/dfs\/dn2 xfs defaults 0 0<\/p>\n
\u52a0\u8f7d
\nmount -a <\/p>\n
\u67e5\u770b<\/p>\n
[root@idc-bigdata-01 ~]# df -hT|grep datas
\n\/dev\/sdb1 xfs 100G 747M 100G 1% \/datas\/dfs\/dn1
\n\/dev\/sdc1 xfs 100G 747M 100G 1% \/datas\/dfs\/dn2<\/p>\n
\u6309\u4ee5\u4e0b\u987a\u5e8f\u542f\u52a8hadoop\u76f8\u5173\u670d\u52a1：<\/p>\n
1.\u542f\u52a8zookeeper\u96c6\u7fa4<\/p>\n
zkServer.sh start
\n\u68c0\u67e5\u542f\u52a8\u60c5\u51b5： jps ，\u6bcf\u4e2a\u8282\u70b9\u5305\u542bQuorumPeerMain\u8fdb\u7a0b\u4ee3\u8868zk\u96c6\u7fa4\u542f\u52a8\u6210\u529f\u6216\u8005\u6bcf\u4e2a\u8282\u70b9\u6267\u884c：
\nzkServer.sh status，\u67e5\u770b\u662f\u5426\u6709leader\u8282\u70b9\u3002<\/p>\n
2.\u6bcf\u4e2anamenode\u8282\u70b9\u4e0a\u542f\u52a8journalnode\u8fdb\u7a0b
\nhdfs –daemon start journalnode<\/p>\n
3.\u5728\u7b2c\u4e00\u53f0namenode\u8282\u70b9\u4e0a\u683c\u5f0f\u5316namenode
\nhdfs namenode -format
\n\u542f\u52a8\u8fd9\u53f0\u8282\u70b9\u7684namenode
\nhdfs –daemon start namenode<\/p>\n
4.\u5176\u4ed6Namenode\u8282\u70b9\u4e0a\u9996\u6b21\u8981\u624b\u52a8\u540c\u6b65\u4e00\u6b21\u6570\u636e，\u5728\u53e6\u5916\u4fe9\u4e2a\u8282\u70b9\u6267\u884c：
\nhdfs namenode -bootstrapStandby
\n\u540c\u6b65\u5b8c\u6210\u540e\u542f\u52a8namenode
\nhdfs –daemon start namenode<\/p>\n
5.\u521d\u59cb\u5316ZKFC \u6ce8\u8fd9\u91cc\u7684\u521d\u59cb\u5316\u4e5f\u53ef\u4ee5\u653e\u5728\u7b2c\u4e00\u6b65\u64cd\u4f5c
\nZKFC\u7528\u4e8e\u76d1\u63a7active namenode\u8282\u70b9\u662f\u5426\u6302\u6389，\u901a\u77e5\u5176\u5b83\u8282\u70b9\u4e0a\u7684ZKFC\u5f3a\u884c\u6740\u6b7b\u81ea\u5df1ZKFC\u8282\u70b9\u4e0a\u7684namenode
\n（\u9632\u6b62\u5176\u5047\u6b7b\u72b6\u6001\u4ea7\u751f\u96c6\u7fa4namenode\u8111\u88c2\u7684\u53d1\u751f），\u7136\u540e\u9009\u4e3e\u51fa\u5176\u4ed6namenode\u4e3aactive\u8282\u70b9
\n\u96c6\u7fa4\u9996\u6b21\u642d\u5efa\u9700\u8981\u5728zookeeper\u4e2d\u521d\u59cb\u5316namenode\u4fe1\u606f，\u5728namenode1\u8282\u70b9\u6267\u884c\u547d\u4ee4：
\nhdfs zkfc -formatZK
\n\u4e3b\u8282\u70b9\u542f\u52a8zkfc
\nhdfs –daemon start zkfc<\/p>\n
#### \u542f\u52a8HDFS\u76f8\u5173\u8fdb\u7a0b
\n \u5728namenode1\u8282\u70b9\u4e0a，\u542f\u52a8HDFS\u6574\u4e2a\u96c6\u7fa4（\u5305\u62ec\u5176\u4ed6\u8282\u70b9）\u7684\u5176\u4ed6\u8fdb\u7a0b（Datanode\/DFSZKFailoverController），\u6267\u884c\u547d\u4ee4：
\nstart-dfs.sh
\nstart-yarn.sh
\n\u6216\u8005\u6267\u884cstart-all.sh <\/p>\n
6.\u67e5\u770bnamenode active\u8282\u70b9<\/p>\n
hdfs haadmin -getAllServiceState
\n[hadoop@idc-bigdata-01 ~]$ hdfs haadmin -getAllServiceState
\nidc-bigdata-01:8020 active
\nidc-bigdata-02:8020 standby
\nidc-bigdata-03:8020 standby <\/p>\n
7. \u542f\u52a8\u65e5\u5fd7\u670d\u52a1
\nidc-bigdata-03
\nmapred –daemon start historyserver<\/p>\n
8. \u9a8c\u8bc1\u9ad8\u53ef\u7528
\nkill -9 active namenode\u8fdb\u7a0b，\u67e5\u770b\u9875\u9762\u72b6\u6001，\u53ef\u53d1\u73b0\u53e6\u5916\u67d0\u4e2anamenode\u81ea\u52a8\u5207\u6362\u6210active\u72b6\u6001\u3002
\n\u8bb0\u4f4fkill\u6389\u7684，\u5b9e\u9a8c\u7ed3\u675f\u540e\u518d\u542f\u52a8\u8d77\u6765\u3002
\nkill \u6216 hdfs –daemon stop namenode
\n\u96c6\u7fa4\u4e00\u5171\u4e00\u6709namenode \u6700\u591a\u53ef\u4ee5\u5bb9\u5fcd\u4e24\u4e2a\u5b95\u673a<\/p>\n
9. \u9a8c\u8bc1\u96c6\u7fa4\u662f\u5426\u53ef\u7528<\/p>\n
hdfs dfs -mkdir \/data
\nhdfs dfs -mkdir hdfs:\/\/idc-bigdata-cluster\/wordcount
\nhdfs dfs -put wordcount.txt hdfs:\/\/idc-bigdata-cluster\/wordcount
\nhdfs dfs -ls hdfs:\/\/idc-bigdata-cluster\/wordcount<\/p>\n
\u683c\u5f0f\u5316ZooKeeper\u96c6\u7fa4，\u76ee\u7684\u662f\u5728ZooKeeper\u96c6\u7fa4\u4e0a\u5efa\u7acbHA\u7684\u76f8\u5e94\u8282\u70b9，\u4efb\u610f\u8282\u70b9\u6267\u884c \u8fd9\u91cc\u4f7f\u7528\u7b2c\u4e00\u4e2a\u8282\u70b9 idc-bigdata-01<\/p>\n
su – hadoop
\nhdfs zkfc -formatZK<\/p>\n
\u6267\u884c\u8f93\u51fa<\/p>\n
2023-06-02 04:45:17,095 INFO ha.ActiveStandbyElector: Session connected.
\n2023-06-02 04:45:17,139 INFO ha.ActiveStandbyElector: Successfully created \/hadoop-ha\/idc-bigdata-cluster in ZK.
\n2023-06-02 04:45:17,251 INFO zookeeper.ZooKeeper: Session: 0x2000134fa350000 closed
\n2023-06-02 04:45:17,252 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2000134fa350000
\n2023-06-02 04:45:17,252 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x2000134fa350000
\n2023-06-02 04:45:17,257 INFO tools.DFSZKFailoverController: SHUTDOWN_MSG:
\n\/************************************************************
\nSHUTDOWN_MSG: Shutting down DFSZKFailoverController at idc-bigdata-01\/10.62.1.192
\n************************************************************\/<\/p>\n
\u9a8c\u8bc1zkfc\u662f\u5426\u683c\u5f0f\u5316\u6210\u529f<\/p>\n
# zkCli.sh<\/p>\n
[zk: localhost:2181(CONNECTED) 1] ls \/
\n[hadoop-ha, zookeeper]
\n[zk: localhost:2181(CONNECTED) 2] ls \/hadoop-ha
\n[idc-bigdata-cluster]<\/p>\n
idc-bigdata-01\u3001idc-bigdata-02\u53caidc-bigdata-03\u8282\u70b9\u542f\u52a8journalnode<\/p>\n
su – hadoop
\nhdfs –daemon start journalnode \u6bcf\u4e2a\u8282\u70b9\u90fd\u9700\u8981\u542f\u52a8<\/p>\n
jps \u67e5\u770bjava\u8fdb\u7a0b<\/p>\n
[hadoop@idc-bigdata-01 ~]$ jps -l
\n46520 org.apache.hadoop.hdfs.qjournal.server.JournalNode<\/p>\n
\u6bcf\u4e2a\u8282\u70b9\u4e0a\u90fd\u6709\u8fd9\u4e2a\u8fdb\u7a0b
\n\u76d1\u542c\u7aef\u53e3\u4e3a：8485<\/p>\n
\u5728\u5176\u4e2d\u4e00\u4e2anamenode\u8282\u70b9\u6267\u884c\u683c\u5f0f\u5316，\u4ee5\u5728idc-bigdata-01\u8282\u70b9\u4e3a\u4f8b<\/p>\n
su – hadoop
\nhdfs namenode -format<\/p>\n
\u6267\u884c\u8f93\u51fa <\/p>\n
2023-06-02 04:59:50,441 INFO util.GSet: VM type = 64-bit
\n2023-06-02 04:59:50,442 INFO util.GSet: 0.029999999329447746% max memory 3.2 GB = 1007.6 KB
\n2023-06-02 04:59:50,442 INFO util.GSet: capacity = 2^17 = 131072 entries
\n2023-06-02 04:59:51,244 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1983029622-10.62.1.192-1685653191244
\n2023-06-02 04:59:51,314 INFO common.Storage: Storage directory \/datas\/dfs\/name has been successfully formatted.
\n2023-06-02 04:59:51,597 INFO namenode.FSImageFormatProtobuf: Saving image file \/datas\/dfs\/name\/current\/fsimage.ckpt_0000000000000000000 using no compression
\n2023-06-02 04:59:51,716 INFO namenode.FSImageFormatProtobuf: Image file \/datas\/dfs\/name\/current\/fsimage.ckpt_0000000000000000000 of size 398 bytes saved in 0 seconds .
\n2023-06-02 04:59:51,745 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
\n2023-06-02 04:59:51,835 INFO namenode.FSNamesystem: Stopping services started for active state
\n2023-06-02 04:59:51,835 INFO namenode.FSNamesystem: Stopping services started for standby state
\n2023-06-02 04:59:51,841 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
\n2023-06-02 04:59:51,842 INFO namenode.NameNode: SHUTDOWN_MSG:
\n\/************************************************************
\nSHUTDOWN_MSG: Shutting down NameNode at idc-bigdata-01\/10.62.1.192
\n************************************************************\/<\/p>\n
\u542f\u52a8idc-bigdata-01\u8282\u70b9nameNode<\/p>\n
su – hadoop
\nhdfs –daemon start namenode<\/p>\n
\u67e5\u770b\u8fdb\u7a0b<\/p>\n
[hadoop@idc-bigdata-01 ~]$ jps
\n47863 NameNode
\n46520 JournalNode
\n47918 Jps<\/p>\n
[hadoop@idc-bigdata-01 ~]$ ls -lh \/datas\/dfs\/name\/
\ntotal 4.0K
\ndrwx—— 2 hadoop hadoop 112 Jun 2 04:59 current
\n-rw-r–r– 1 hadoop hadoop 20 Jun 2 05:06 in_use.lock<\/p>\n
\u5c06idc-bigdata-01\u8282\u70b9\u4e0anamenode\u7684\u6570\u636e\u540c\u6b65\u5230\u5176\u4ed6nameNode\u8282\u70b9，\u5728idc-bigdata-02\u3001idc-bigdata-03\u8282\u70b9\u6267\u884c：<\/p>\n
hdfs namenode -bootstrapStandby<\/p>\n
\u6267\u884c\u8fc7\u7a0b<\/p>\n
idc-bigdata-02<\/p>\n
=====================================================
\n2023-06-01 21:13:27,641 INFO common.Storage: Storage directory \/datas\/dfs\/name has been successfully formatted.
\n2023-06-01 21:13:27,694 INFO namenode.FSEditLog: Edit logging is async:true
\n2023-06-01 21:13:27,776 INFO namenode.TransferFsImage: Opening connection to http:\/\/idc-bigdata-01:9870\/imagetransfer?getimage=1&txid=0&storageInfo=-66:53130945:1685653191244:CID-c35ed3c1-4131-4a48-bf1b-7d8ffe34a40d&bootstrapstandby=true
\n2023-06-01 21:13:27,877 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB\/s. Synchronous (fsync) write to disk of \/datas\/dfs\/name\/current\/fsimage.ckpt_0000000000000000000 took 0.00s.
\n2023-06-01 21:13:27,878 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 398 bytes.
\n2023-06-01 21:13:27,885 INFO ha.BootstrapStandby: Skipping InMemoryAliasMap bootstrap as it was not configured
\n2023-06-01 21:13:27,888 INFO namenode.NameNode: SHUTDOWN_MSG:
\n\/************************************************************
\nSHUTDOWN_MSG: Shutting down NameNode at idc-bigdata-02\/10.62.1.193<\/p>\n
idc-bigdata-03
\n=====================================================
\n2023-06-02 05:00:58,927 INFO common.Storage: Storage directory \/datas\/dfs\/name has been successfully formatted.
\n2023-06-02 05:00:58,971 INFO namenode.FSEditLog: Edit logging is async:true
\n2023-06-02 05:00:59,045 INFO namenode.TransferFsImage: Opening connection to http:\/\/idc-bigdata-01:9870\/imagetransfer?getimage=1&txid=0&storageInfo=-66:53130945:1685653191244:CID-c35ed3c1-4131-4a48-bf1b-7d8ffe34a40d&bootstrapstandby=true
\n2023-06-02 05:00:59,059 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB\/s. Synchronous (fsync) write to disk of \/datas\/dfs\/name\/current\/fsimage.ckpt_0000000000000000000 took 0.00s.
\n2023-06-02 05:00:59,059 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 398 bytes.
\n2023-06-02 05:00:59,077 INFO ha.BootstrapStandby: Skipping InMemoryAliasMap bootstrap as it was not configured
\n2023-06-02 05:00:59,081 INFO namenode.NameNode: SHUTDOWN_MSG:
\n\/************************************************************
\nSHUTDOWN_MSG: Shutting down NameNode at idc-bigdata-03\/10.62.1.194
\n************************************************************\/<\/p>\n
\u542f\u52a8idc-bigdata-02\u53caidc-bigdata-03\u8282\u70b9nameNode<\/p>\n
hdfs –daemon start namenode<\/p>\n
\u67e5\u770b\u8fdb\u7a0b<\/p>\n
======================idc-bigdata-01======================
\n47863 NameNode
\n46520 JournalNode<\/p>\n
======================idc-bigdata-02======================<\/p>\n
21208 NameNode
\n19946 JournalNode
\n======================idc-bigdata-03======================
\n19938 JournalNode
\n21182 NameNode<\/p>\n
\u76d1\u63a7\u7aef\u53e3 8020 9870<\/p>\n
\u6d4f\u89c8\u5668\u8bbf\u95eeNameNode,\u5f53\u524d\u6240\u6709NameNode\u90fd\u662fstandby\u72b6\u6001：<\/p>\n
http:\/\/idc-bigdata-01:9870\/
\nhttp:\/\/idc-bigdata-02:9870\/
\nhttp:\/\/idc-bigdata-03:9870\/<\/p>\n
start-all.sh<\/p>\n
\u6b64\u65f6\u518d\u6b21\u67e5\u770bnameNode\u754c\u9762，\u53d1\u73b0\u5df2\u7ecf\u9009\u4e3e\u51fa\u4e00\u4e2aactive\u8282\u70b9:<\/p>\n
http:\/\/idc-bigdata-01:9870\/<\/p>\n
======================idc-bigdata-01======================<\/p>\n
52096 NameNode
\n53428 ResourceManager
\n52949 DFSZKFailoverController
\n52310 DataNode
\n53646 NodeManager
\n52638 JournalNode<\/p>\n
======================idc-bigdata-02======================<\/p>\n
23856 NameNode
\n24962 NodeManager
\n24514 DFSZKFailoverController
\n24051 DataNode
\n24775 ResourceManager
\n24269 JournalNode<\/p>\n
======================idc-bigdata-03======================<\/p>\n
24178 DFSZKFailoverController
\n24469 NodeManager
\n23931 JournalNode
\n23723 DataNode
\n23532 NameNode<\/p>\n
[hadoop@idc-bigdata-01 ~]$ hdfs dfs -put .bash_history \/
\n[hadoop@idc-bigdata-01 ~]$ hdfs dfs -ls \/
\nFound 12 items
\n-rw-r–r– 3 hadoop supergroup 7818 2023-06-02 10:23 \/.bash_history<\/p>\n
\u8ba1\u7b97\u5706\u5468\u7387<\/p>\n
[hadoop@idc-bigdata-01 ~]$ hadoop jar \/opt\/module\/hadoop\/share\/hadoop\/mapreduce\/hadoop-mapreduce-examples-3.3.5.jar pi 10 100<\/p>\n
……
\nJob Finished in 3.059 seconds
\nEstimated value of Pi is 3.14800000000000000000<\/p>\n
echo "hello hadoop" > wordtest
\nhadoop fs -put wordtest \/wordtest<\/p>\n
\u6267\u884c mapreduce \u540e\u751f\u6210\u7ed3\u679c\u6587\u4ef6
\nhadoop jar \/opt\/module\/hadoop\/share\/hadoop\/mapreduce\/hadoop-mapreduce-examples-3.3.5.jar wordcount \/wordtest \/result<\/p>\n
\u67e5\u770b\u7edf\u8ba1\u7ed3\u679c<\/p>\n
hadoop fs -cat \/result\/part-r-00000<\/p>\n
[hadoop@idc-bigdata-01 ~]$ hadoop fs -cat \/result\/part-r-00000
\nhadoop1
\nhello1<\/p>\n
\u6d4b\u8bd5\u662f\u5426\u80fd\u591f\u5b8c\u6210\u81ea\u52a8\u6545\u969c\u8f6c\u79fb\u3002<\/p>\n
\u4e4b\u540e\u5237\u65b0\u9875\u9762\u6211\u4eec\u53d1\u73b0idc-bigdata-02\u8282\u70b9（\u539fstandy）\u81ea\u52a8\u53d8\u6210\u4e86 active namenode\u3002
\n\u67e5\u770bnodename\u72b6\u6001<\/p>\n
[hadoop@idc-bigdata-01 ~]$ hdfs haadmin -getAllServiceState
\nidc-bigdata-01:8020 standby
\nidc-bigdata-02:8020 active
\nidc-bigdata-03:8020 standby <\/p>\n
\u505c\u6b62idc-bigdata-02 \u4e0anamenode\u8fdb\u7a0b<\/p>\n
\u5728idc-bigdata-02\u8282\u70b9active namenode\u4e0a\u6267\u884c jps ，\u786e\u5b9anamenode\u8fdb\u7a0b，kill \u5c06\u5176\u6740\u6389
\nkill -9 2424 \u6216 hdfs –daemon stop namenode<\/p>\n
[hadoop@idc-bigdata-02 ~]$ hdfs –daemon stop namenode
\n[hadoop@idc-bigdata-02 ~]$ jps
\n60886 Jps
\n53494 DataNode
\n54248 ResourceManager
\n53706 JournalNode
\n54443 NodeManager
\n53918 DFSZKFailoverController<\/p>\n
\u67e5\u770bnamenode\u72b6\u6001<\/p>\n
[hadoop@idc-bigdata-02 ~]$ hdfs haadmin -getAllServiceState
\nidc-bigdata-01:8020 standby
\n2023-06-02 11:37:47,527 INFO ipc.Client: Retrying connect to server: idc-bigdata-02\/10.62.1.193:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
\nidc-bigdata-02:8020 Failed to connect: Call From idc-bigdata-02\/10.62.1.193 to idc-bigdata-02:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http:\/\/wiki.apache.org\/hadoop\/ConnectionRefused
\nidc-bigdata-03:8020 active
\n[hadoop@idc-bigdata-02 ~]$ hdfs haadmin -getServiceState nn1<\/p>\n
\u73b0\u5728namenode \u5df2\u5207\u6362\u5230idc-bigdata-03 <\/p>\n
\u518d\u6b21\u9a8c\u8bc1\u4e0a\u4f20\u4e0b\u8f7d\u7b49\u529f\u80fd\u662f\u5426\u6b63\u5e38<\/p>\n
HA\u6a21\u5f0fRM\u7684active\u8282\u70b9<\/p>\n
yarn rmadmin -getAllServiceState
\nyarn rmadmin -getServiceState rm1<\/p>\n
\u5207\u6362
\nyarn rmadmin -transitionToStandby rm1<\/p>\n
\u68c0\u67e5hadoop\u96c6\u7fa4\u72b6\u6001，\u5982\u679c\u96c6\u7fa4\u4e2d\u5b58\u5728hadoop\u7684\u670d\u52a1\u8fdb\u7a0b，\u5c06hadoop\u7684\u6240\u6709\u670d\u52a1\u7ec4\u4ef6\u90fd\u5173\u95ed \u5173\u95edhadoop\u96c6\u7fa4\u670d\u52a1\u7ec4\u4ef6 \u6e05\u7406hadoop\u96c6\u7fa4\u5b89\u88c5\u76ee\u5f55\u4e0b\u7684data\u548clogs\u76ee\u5f55<\/p>\n
\u505c\u4e0azookeeper \u96c6\u7fa4，\u5220\u9664 zookeeper \u76f8\u5173\u5220\u9664 rm -rf \/data\/zookeeper\/data\/version-2 rm -rf \/datas\/dfs\/dn1\/ rm -rf \/datas\/dfs\/dn2\/* rm -rf \/datas\/dfs\/jn rm -rf \/datas\/dfs\/nn rm -rf \/datas\/dfs\/tmp<\/p>\n
1.\u542f\u52a8zookeeper\u96c6\u7fa4<\/p>\n
zkServer.sh start
\n\u68c0\u67e5\u542f\u52a8\u60c5\u51b5： jps ，\u6bcf\u4e2a\u8282\u70b9\u5305\u542bQuorumPeerMain\u8fdb\u7a0b\u4ee3\u8868zk\u96c6\u7fa4\u542f\u52a8\u6210\u529f\u6216\u8005\u6bcf\u4e2a\u8282\u70b9\u6267\u884c：
\nzkServer.sh status，\u67e5\u770b\u662f\u5426\u6709leader\u8282\u70b9\u3002<\/p>\n
2.\u6bcf\u4e2anamenode\u8282\u70b9\u4e0a\u542f\u52a8journalnode\u8fdb\u7a0b
\nhdfs –daemon start journalnode<\/p>\n
3.\u5728\u7b2c\u4e00\u53f0namenode\u8282\u70b9\u4e0a\u683c\u5f0f\u5316namenode
\nhdfs namenode -format
\n\u542f\u52a8\u8fd9\u53f0\u8282\u70b9\u7684namenode
\nhdfs –daemon start namenode<\/p>\n
4.\u5176\u4ed6Namenode\u8282\u70b9\u4e0a\u9996\u6b21\u8981\u624b\u52a8\u540c\u6b65\u4e00\u6b21\u6570\u636e，\u5728\u53e6\u5916\u4fe9\u4e2a\u8282\u70b9\u6267\u884c：
\nhdfs namenode -bootstrapStandby
\n\u540c\u6b65\u5b8c\u6210\u540e\u542f\u52a8namenode
\nhdfs –daemon start namenode<\/p>\n
5.\u521d\u59cb\u5316ZKFC \u6ce8\u8fd9\u91cc\u7684\u521d\u59cb\u5316\u4e5f\u53ef\u4ee5\u653e\u5728\u7b2c\u4e00\u6b65\u64cd\u4f5c
\nZKFC\u7528\u4e8e\u76d1\u63a7active namenode\u8282\u70b9\u662f\u5426\u6302\u6389，\u901a\u77e5\u5176\u5b83\u8282\u70b9\u4e0a\u7684ZKFC\u5f3a\u884c\u6740\u6b7b\u81ea\u5df1ZKFC\u8282\u70b9\u4e0a\u7684namenode
\n（\u9632\u6b62\u5176\u5047\u6b7b\u72b6\u6001\u4ea7\u751f\u96c6\u7fa4namenode\u8111\u88c2\u7684\u53d1\u751f），\u7136\u540e\u9009\u4e3e\u51fa\u5176\u4ed6namenode\u4e3aactive\u8282\u70b9
\n\u96c6\u7fa4\u9996\u6b21\u642d\u5efa\u9700\u8981\u5728zookeeper\u4e2d\u521d\u59cb\u5316namenode\u4fe1\u606f，\u5728namenode1\u8282\u70b9\u6267\u884c\u547d\u4ee4：
\nhdfs zkfc -formatZK
\n\u4e3b\u8282\u70b9\u542f\u52a8zkfc
\nhdfs –daemon start zkfc<\/p>\n
#### \u542f\u52a8HDFS\u76f8\u5173\u8fdb\u7a0b
\n \u5728namenode1\u8282\u70b9\u4e0a，\u542f\u52a8HDFS\u6574\u4e2a\u96c6\u7fa4（\u5305\u62ec\u5176\u4ed6\u8282\u70b9）\u7684\u5176\u4ed6\u8fdb\u7a0b（Datanode\/DFSZKFailoverController），\u6267\u884c\u547d\u4ee4：
\nstart-dfs.sh
\nstart-yarn.sh
\n\u6216\u8005\u6267\u884cstart-all.sh <\/p>\n
6.\u67e5\u770bnamenode active\u8282\u70b9<\/p>\n
hdfs haadmin -getAllServiceState
\n[hadoop@idc-bigdata-01 ~]$ hdfs haadmin -getAllServiceState
\nidc-bigdata-01:8020 active
\nidc-bigdata-02:8020 standby
\nidc-bigdata-03:8020 standby <\/p>\n
7. \u542f\u52a8\u65e5\u5fd7\u670d\u52a1
\nidc-bigdata-03
\nmapred –daemon start historyserver<\/p>\n
8. \u9a8c\u8bc1\u9ad8\u53ef\u7528
\nkill -9 active namenode\u8fdb\u7a0b，\u67e5\u770b\u9875\u9762\u72b6\u6001，\u53ef\u53d1\u73b0\u53e6\u5916\u67d0\u4e2anamenode\u81ea\u52a8\u5207\u6362\u6210active\u72b6\u6001\u3002
\n\u8bb0\u4f4fkill\u6389\u7684，\u5b9e\u9a8c\u7ed3\u675f\u540e\u518d\u542f\u52a8\u8d77\u6765\u3002
\nkill \u6216 hdfs –daemon stop namenode
\n\u96c6\u7fa4\u4e00\u5171\u4e00\u6709namenode \u6700\u591a\u53ef\u4ee5\u5bb9\u5fcd\u4e24\u4e2a\u5b95\u673a<\/p>\n
9. \u9a8c\u8bc1\u96c6\u7fa4\u662f\u5426\u53ef\u7528<\/p>\n
hdfs dfs -mkdir \/data
\nhdfs dfs -mkdir hdfs:\/\/idc-bigdata-cluster\/wordcount
\nhdfs dfs -put wordcount.txt hdfs:\/\/idc-bigdata-cluster\/wordcount
\nhdfs dfs -ls hdfs:\/\/idc-bigdata-cluster\/wordcount<\/p>\n
\u67e5\u770bnamenode \u76ee\u5f55\u7ed3\u6784<\/p>\n
ls -lh \/datas\/dfs\/nn\/current\/<\/p>\n
-rw-rw-r– 1 hadoop hadoop 42 Jun 4 11:51 edits_0000000000000002362-0000000000000002363
\n-rw-rw-r– 1 hadoop hadoop 1.0M Jun 4 11:51 edits_inprogress_0000000000000002364
\n-rw-rw-r– 1 hadoop hadoop 9.0K Jun 4 10:28 fsimage_0000000000000002018
\n-rw-rw-r– 1 hadoop hadoop 62 Jun 4 10:28 fsimage_0000000000000002018.md5
\n-rw-rw-r– 1 hadoop hadoop 12K Jun 4 11:28 fsimage_0000000000000002263
\n-rw-rw-r– 1 hadoop hadoop 62 Jun 4 11:28 fsimage_0000000000000002263.md5
\n-rw-rw-r– 1 hadoop hadoop 5 Jun 4 11:51 seen_txid
\n-rw-r–r– 1 hadoop hadoop 215 Jun 3 21:14 VERSION<\/p>\n
fsimage\u662fNameNode\u5143\u6570\u636e\u5728\u5185\u5b58\u6ee1\u4e86\u540e，\u6301\u4e45\u5316\u4fdd\u5b58\u5230\u7684\u6587\u4ef6\u3002 fsimage*.md5 \u662f\u6821\u9a8c\u6587\u4ef6，\u7528\u4e8e\u6821\u9a8cfsimage\u7684\u5b8c\u6574\u6027\u3002 seen_txid \u662fhadoop\u7684\u7248\u672c vession\u6587\u4ef6\u91cc\u4fdd\u5b58 namespaceID：NameNode\u7684\u552f\u4e00ID clusterID:\u96c6\u7fa4ID NameNode\u548cDataNode\u7684\u96c6\u7fa4ID\u5e94\u8be5\u4e00\u81f4，\u8868\u660e\u662f\u4e00\u4e2a\u96c6\u7fa4<\/p>\n
[root@idc-bigdata-01 ~]# more \/datas\/dfs\/nn\/current\/VERSION #Sat Jun 03 21:14:52 CST 2023 namespaceID=1498354829 clusterID=CID-24440b17-58ee-4d55-a589-ea0a5c5c0be6 cTime=1685798092376 storageType=NAME_NODE blockpoolID=BP-899285325-10.62.1.192-1685798092376 layoutVersion=-66<\/p>\n
\u6ce8\u610f：<\/p>\n
\u5982\u679c\u9700\u8981\u91cd\u65b0\u683c\u5f0f\u5316NameNode,\u9700\u8981\u5148\u5c06\u539f\u6765NameNode\u548cDataNode\u4e0b\u7684\u6587\u4ef6\u5168\u90e8\u5220\u9664，\u4e0d\u7136\u4f1a\u62a5\u9519，NameNode\u548cDataNode\u6240\u5728 \u76ee\u5f55\u662f\u5728core-site.xml\u4e2dhadoop.tmp.dir\u3001dfs.namenode.name.dir\u3001dfs.datanode.data.dir\u5c5e\u6027\u914d\u7f6e\u7684\u3002<\/p>\n
\u56e0\u4e3a\u6bcf\u6b21\u683c\u5f0f\u5316，\u9ed8\u8ba4\u662f\u521b\u5efa\u4e00\u4e2a\u96c6\u7fa4ID，\u5e76\u5199\u5165NameNode\u548cDataNode\u7684VERSION\u6587\u4ef6\u4e2d（VERSION\u6587\u4ef6\u6240\u5728\u76ee\u5f55\u4e3a dfs\/name\/current \u548c dfs\/data\/current），\u91cd\u65b0\u683c\u5f0f\u5316\u65f6，\u9ed8\u8ba4\u4f1a\u751f\u6210\u4e00\u4e2a\u65b0\u7684\u96c6\u7fa4ID,\u5982\u679c\u4e0d\u5220\u9664\u539f\u6765\u7684\u76ee\u5f55， \u4f1a\u5bfc\u81f4namenode\u4e2d\u7684VERSION\u6587\u4ef6\u4e2d\u662f\u65b0\u7684\u96c6\u7fa4ID,\u800cDataNode\u4e2d\u662f\u65e7\u7684\u96c6\u7fa4ID，\u4e0d\u4e00\u81f4\u65f6\u4f1a\u62a5\u9519\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"
\u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb880\u6b21\uff0c\u70b9\u8d5e20\u6b21\uff0c\u6536\u85cf17\u6b21\u3002Hadoop\u662f\u4e00\u4e2a\u7531Apache\u57fa\u91d1\u4f1a\u6240\u5f00\u53d1\u7684\u5206\u5e03\u5f0f\u7cfb\u7edf\u57fa\u7840\u67b6\u6784\u3002\u7528\u6237\u53ef\u4ee5\u5728\u4e0d\u4e86\u89e3\u5206\u5e03\u5f0f\u5e95\u5c42\u7ec6\u8282\u7684\u60c5\u51b5\u4e0b\uff0c\u5f00\u53d1\u5206\u5e03\u5f0f\u7a0b\u5e8f\u3002\u5145\u5206\u5229\u7528\u96c6\u7fa4\u7684\u5a01\u529b\u8fdb\u884c\u9ad8\u901f\u8fd0\u7b97\u548c\u5b58\u50a8\u3002HADOOP\u96c6\u7fa4\u5177\u4f53\u6765\u8bf4\u5305\u542b\u4e24\u4e2a\u96c6\u7fa4\uff1aHDFS\u96c6\u7fa4\u548cYARN\u96c6\u7fa4\uff0c\u4e24\u8005\u903b\u8f91\u4e0a\u5206\u79bb\uff0c\u4f46\u7269\u7406\u4e0a\u5e38\u5728\u4e00\u8d77\u3002HDFS\u96c6\u7fa4\u8d1f\u8d23\u6d77\u91cf\u6570\u636e\u7684\u5b58\u50a8\uff0c\u96c6\u7fa4\u4e2d\u7684\u89d2\u8272\u4e3b\u8981\u6709\uff1aNameNode\u3001DataNode\u3001SecondaryNameNodeYARN\u96c6\u7fa4\u8d1f\u8d23\u6d77\u91cf\u6570\u636e\u8fd0\u7b97\u65f6\u7684\u8d44\u6e90\u8c03\u5ea6\uff0c\u96c6\u7fa4\u4e2d\u7684\u89d2\u8272\u4e3b\u8981\u6709\uff1aResourceManager\u3001NodeManager\u3002_\u57fa\u4e8ekylin\u7684hadoop\u96c6\u7fa4\u642d\u5efa<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[779,323,43,44],"topic":[],"class_list":["post-16056","post","type-post","status-publish","format-standard","hentry","category-server","tag-hadoop","tag-323","tag-43","tag-44"],"yoast_head":"\n