有个小技巧：先配置好一个，在（宿主机上）复制scp -r拷贝Spark到其他Slaves。
(资料图片)
【在test-cluster-hap-master-01虚拟主机上】
将已下载好的Spark压缩包（spark-3.1.1-bin-hadoop-3.2.2-lbx-jszt.tgz）通过工具【XFtp】拷贝到虚拟主机的opt目录下：
2.通过脚本挂起镜像cd /opt/script/setup/spark#!/bin/bash #编写作者：千羽的编程时光cname="test-cluster-spk-master-01"#port1="8080"#port2="7077"log="/opt/data/"${cname}images="10.249.0.137:80/base/jdk-1.8:20210202"mkdir -p ${log}mkdir ${log}/logsmkdir ${log}/workmkdir ${log}/datamkdir ${log}/jars# docker run -d --net=overlay-net --ip ${ip} -p ${port1}:${port1} -p ${port2}:${port2} --name ${cname} --hostname ${cname} --privileged=true --restart=always docker run -d --net=host --name ${cname} --hostname ${cname} --privileged=true --restart=always \-v ${log}/logs:/usr/local/spark-3.1.1/logs \-v ${log}/work:/usr/local/spark-3.1.1/work \-v ${log}/jars:/usr/local/spark-3.1.1/jars \-v ${log}/data:/opt/data \${images} \/usr/sbin/init#!/bin/bash cname="test-cluster-spk-master-02"#port1="8080"#port2="7077"log="/opt/data/"${cname}images="10.249.0.137:80/base/jdk-1.8:20210202"mkdir -p ${log}mkdir ${log}/logsmkdir ${log}/workmkdir ${log}/datamkdir ${log}/jars#docker run -d --net=overlay-net --ip ${ip} -p ${port1}:${port1} -p ${port2}:${port2} --name ${cname} --hostname ${cname} --privileged=true --restart=always docker run -d --net=host --name ${cname} --hostname ${cname} --privileged=true --restart=always \-v ${log}/logs:/usr/local/spark-3.1.1/logs \-v ${log}/work:/usr/local/spark-3.1.1/work \-v ${log}/jars:/usr/local/spark-3.1.1/jars \-v ${log}/data:/opt/data \${images} \/usr/sbin/init#!/bin/bash cname="test-cluster-spk-slave-01"#port1="8080"#port2="7077"log="/opt/data/"${cname}images="10.249.0.137:80/base/jdk-1.8:20210202"mkdir -p ${log}mkdir ${log}/logsmkdir ${log}/workmkdir ${log}/datamkdir ${log}/jars#docker run -d --net=overlay-net --ip ${ip} -p ${port1}:${port1} -p ${port2}:${port2} --name ${cname} --hostname ${cname} --privileged=true --restart=always docker run -d --net=host --name ${cname} --hostname ${cname} --privileged=true --restart=always \-v ${log}/logs:/usr/local/spark-3.1.1/logs \-v ${log}/work:/usr/local/spark-3.1.1/work \-v ${log}/jars:/usr/local/spark-3.1.1/jars \-v ${log}/data:/opt/data \${images} \/usr/sbin/init[root@zookeeper-03-test spark]# ll总用量 4-rw-r--r--. 1 root root 1166 7月 28 17:44 install.sh[root@zookeeper-03-test spark]# chmod +x install.sh [root@zookeeper-03-test spark]# ll总用量 4-rwxr-xr-x. 1 root root 1166 7月 28 17:44 install.sh[root@zookeeper-03-test spark]#在容器映射目录下：/opt/data/test-cluster-spk-slave-01/data
[root@hadoop-01 data]# pwd/opt/data用Xftp上传包
这里需要上传两个，使用的是spark-3.1.1-bin-without-hadoop.tgz
但是需要将spark-3.1.1-bin-hadoop-3.2.2-lbx-jszt下的jars包移到/usr/local/spark-3.1.1/jars下
4.解压安装包mkdir -p /usr/local/spark-3.1.1cd /opt/datatar -zxvf spark-3.1.1-bin-without-hadoop.tgz -C /usr/local/spark-3.1.1编辑全局变量
vim /etc/profile增加以下全局变量
export SPARK_HOME=/usr/local/spark-3.1.1 export PATH=$PATH:$SPARK_HOME/bin即时生效
source /etc/profilecd /usr/local/spark-3.1.1/confcp spark-env.sh.template spark-env.shvim spark-env.shexport SPARK_MASTER_IP=test-cluster-spk-master-01export SPARK_WORKER_CORES=1export SPARK_WORKER_MEMORY=800m#export SPARK_DRIVER_MEMORY=4gexport SPARK_EXECUTOR_INSTANCES=2export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoopexport SPARK_LOCAL_DIRS=/home/hadoop/tmp/spark/tmp#定时清理worker文件 一天一次export SPARK_WORKER_OPTS=" -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=86400 -Dspark.worker.cleanup.appDataTtl=86400"export JAVA_HOME=/usr/local/jdk1.8export HADOOP_HOME=/usr/local/hadoopexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport SCALA_HOME=/usr/local/scalaexport PATH=${SCALA_HOME}/bin:$PATHexport SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=zookeeper-01-test:2181,zookeeper-02-test:2181,zookeeper-03-test:2181 -Dspark.deploy.zookeeper.dir=/usr/local/spark"（4）配置workers
cp workers.template workersvim workers# 添加test-cluster-spk-slave-001cp log4j.properties.template log4j.propertiesvim log4j.propertieslog4j.rootCategory=WARN, console（宿主机上）复制scp -r拷贝Spark到其他Slaves节点：
scp -r /usr/local/spark/spark-2.1.0-bin-hadoop2.7 root@slave-001-spark-dev:/usr/local/spark/scp -r /usr/local/spark/spark-2.1.0-bin-hadoop2.7 root@slave-002-spark-dev:/usr/local/spark/scp -r /usr/local/spark/spark-2.1.0-bin-hadoop2.7 root@slave-003-spark-dev:/usr/local/spark/如执行命令出现出现问题时，请现在相应的Slave节点执行mkdir -p /usr/local/spark
复制到master-02时，使用start-mater.sh启动master-02
8.启动spark先启动两个master，然后启动slave节点[root@test-cluster-spk-master-01 sbin]# ./start-master.sh starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-3.1.1/logs/spark-root-org.apache.spark.deploy.master.Master-1-test-cluster-spk-master-01.out[root@test-cluster-spk-master-01 sbin]# jps548 Jps492 Master[root@test-cluster-spk-master-01 sbin]# pwd/usr/local/spark-3.1.1/sbin[root@test-cluster-spk-master-01 sbin]#[root@test-cluster-spk-master-02 sbin]# ./start-master.sh starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-3.1.1/logs/spark-root-org.apache.spark.deploy.master.Master-1-test-cluster-spk-master-02.out[root@test-cluster-spk-master-02 sbin]# pwd/usr/local/spark-3.1.1/sbin[root@test-cluster-spk-master-02 sbin]# jps274 Jps218 Master[root@test-cluster-spk-master-02 sbin]#/usr/local/spark-3.1.1/sbin/start-slave.sh test-cluster-hap-slave-001 test-cluster-hap-master-02:7077,test-cluster-hap-master-02:7077原本是访问http://10.8.46.35:8080 就可，但是我这里在配置镜像的时候，多了8080，导致这里访问不了。看日志可以知道，已经走向8081
所以http://10.8.46.35:8081/即可
主节点1 | 停掉主节点 |
主节点2 | 从节点成为ALIVE |
从节点1 | 从节点1 |
10.遇到的坑包不兼容
这里遇到了许多问题，第一个是包不兼容，导致搭建两次失败
然后换了官方的包spark-3.1.1-bin-without-hadoop，启动还是有问题。
最后通过替换jars才成功。（使用spark-3.1.1-bin-hadoop-3.2.2-lbx-jszt下的jars）
ctrl + p + q 从容器退出到宿主机done~
关键词：
