Kafka集群部署(含开机自启动)

2022-08-31 13:18:47

1.Zookeeper集群部署

1.1 下载安装包并解压

从官网下载所需版本，将下载好后的安装包上传到服务器任一节点上，并解压

[root@kafka-01 ~]# tar -zxf zookeeper-3.5.5.tar.gz -C /opt/software/[root@kafka-01 ~]# cd /opt/software/zookeeper-3.5.5

1.2 修改配置文件

[root@kafka-01 zookeeper-3.5.5]# cp conf/zoo_sample.cfg conf/zoo.cfg[root@kafka-01 zookeeper-3.5.5]# vim conf/zoo.cfg# The number of milliseconds of each tick
tickTime=2000# The number of ticks that the initial# synchronization phase can take
initLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgement
syncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.
dataDir=/var/log/zookeeper/zkData
dataLogDir=/var/log/zookeeper/zkLog# the port at which the clients will connect
clientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1#这改成你自己的主机名称或者IP地址，别傻了吧唧复制粘贴也不改的，这句注释也别留在配置文件里的
server.1=kafka-01:2888:3888
server.2=kafka-02:2888:3888
server.3=kafka-03:2888:3888

dataDir是Zookeeper保存数据的目录,默认情况下Zookeeper将写数据的日志文件也保存在这个目录里；

clientPort是客户端连接Zookeeper服务器的端口,Zookeeper会监听这个端口接受客户端的访问请求；

server.A=B:C:D中的A是一个数字,表示这个是第几号服务器,B是这个服务器的IP地址，C用来集群成员的信息交换,表示这个服务器与集群中的leader服务器交换信息的端口，D是在leader挂掉时专门用来进行选举leader所用的端口。

1.3 创建存储日志目录

每个ZK节点都要创建，创建的名称和配置文件里的得对应

[root@kafka-01 conf]# mkdir -p /vat/log/zookeeper/{zkData,zkLog}

1.4 创建serverID标识

在server.1执行

echo"1">/opt/zookeeper/zkData/myid

在server.2执行

echo"2">/opt/zookeeper/zkData/myid

在server.3执行

echo"3">/opt/zookeeper/zkData/myid

以此类推

2.Kafka集群部署

2.1 下载安装包并解压

从官网下载所需版本，将下载好后的安装包上传到服务器任一节点上，并解压

[root@kafka-01 ~]# tar -zxf kafka_2.11-1.1.0.tgz -C /opt/software/[root@kafka-01 ~]# cd /opt/software/kafka_2.11-1.1.0[root@kafka-01 kafka_2.11-1.1.0]# ll config
total 64
-rw-r--r-- 1 root root  906 Mar 24  2018 connect-console-sink.properties
-rw-r--r-- 1 root root  909 Mar 24  2018 connect-console-source.properties
-rw-r--r-- 1 root root 5807 Mar 24  2018 connect-distributed.properties
-rw-r--r-- 1 root root  883 Mar 24  2018 connect-file-sink.properties
-rw-r--r-- 1 root root  881 Mar 24  2018 connect-file-source.properties
-rw-r--r-- 1 root root 1111 Mar 24  2018 connect-log4j.properties
-rw-r--r-- 1 root root 2730 Mar 24  2018 connect-standalone.properties
-rw-r--r-- 1 root root 1221 Mar 24  2018 consumer.properties
-rw-r--r-- 1 root root 4727 Mar 24  2018 log4j.properties
-rw-r--r-- 1 root root 1919 Mar 24  2018 producer.properties
-rw-r--r-- 1 root root 6890 May 13 10:15 server.properties
-rw-r--r-- 1 root root 1032 Mar 24  2018 tools-log4j.properties
-rw-r--r-- 1 root root 1023 Mar 24  2018 zookeeper.properties

2.2 修改配置文件

演示只是修改了部分配置信息，可根据需求添加特定的配置信息，配置信息参考Kafka Configuration

[root@kafka-01 kafka_2.11-1.1.0]# vim config/server.properties# The id of the broker. This must be set to a unique integer for each broker.#broker.id每个节点需要设置不同的值
broker.id=0# A comma separated list of directories under which to store log files#配置kafka日志存储路径，包括kafka的数据日志，要是有多个目录用逗号间隔就行
log.dirs=/var/log/kafka# Zookeeper connection string (see zookeeper docs for details).# This is a comma separated host:port pairs, each corresponding to a zk# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".# You can also append an optional chroot string to the urls to specify the# root directory for all kafka znodes.# 配置ZK集群的IP:PORT
zookeeper.connect=kafka-01:2181,kafka-02:2181,kafka-03:2181

2.3 分发到各个节点

同步方式很多，为了演示用scp分发，也可以使用其他的方式

[root@kafka-01 kafka_2.11-1.1.0]# scp -r /opt/software/kafka_2.11-1.1.0 root@kafka-02:/opt/software[root@kafka-01 kafka_2.11-1.1.0]# scp -r /opt/software/kafka_2.11-1.1.0 root@kafka-03:/opt/software

3.配置ZK开机自启动

每个zk节点都要执行以下的操作

3.1 修改Zookeeper启动脚本

开启ZK四字命令(Four Letter Words)，有两种方法，任选一种即可
第一种方法：在启动脚本中添加ZOOMAIN="-Dzookeeper.4lw.commands.whitelist=* ${ZOOMAIN}"，需要注意添加位置确保加载顺序，我添加的位置只是一个参考
第二种方法：在zoo.cfg中添加4lw.commands.whitelist=*

[root@kafka-01 ~]# vim /opt/software/zookeeper-3.5.5/bin/zkServer.shfiif["x$JMXLOG4J"="x"]then
      JMXLOG4J=truefiecho"ZooKeeper remote JMX Port set to$JMXPORT">&2echo"ZooKeeper remote JMX authenticate set to$JMXAUTH">&2echo"ZooKeeper remote JMX ssl set to$JMXSSL">&2echo"ZooKeeper remote JMX log4j set to$JMXLOG4J">&2
    ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=$JMXPORT -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH -Dcom.sun.management.jmxremote.ssl=$JMXSSL -Dzookeeper.jmx.log4j.disable=$JMXLOG4J org.apache.zookeeper.server.quorum.QuorumPeerMain"fielseecho"JMX disabled by user request">&2
    ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"fi#新增变量
ZOOMAIN="-Dzookeeper.4lw.commands.whitelist=*${ZOOMAIN}"if["x$SERVER_JVMFLAGS"!="x"]then
    JVMFLAGS="$SERVER_JVMFLAGS$JVMFLAGS"fiif["x$2"!="x"]then
    ZOOCFG="$ZOOCFGDIR/$2"fi# if we give a more complicated path to the config, don't screw around in $ZOOCFGDIR

3.2 添加Zookeeper服务

注意根据自己的安装位置和版本修改，别傻了吧唧的复制粘贴

[root@kafka-01 conf]# vim /etc/init.d/zookeeper#!/bin/bashexport JAVA_HOME=/opt/software/jdk1.8.0_161/export PATH=$JAVA_HOME/bin:$PATH#chkconfig:2345 20 90#description:zookeeper#processname:zookeepercase$1in
        start)
        /opt/software/zookeeper-3.5.5/bin/zkServer.sh start;;
        stop)
        /opt/software/zookeeper-3.5.5/bin/zkServer.sh stop;;
        status)
        /opt/software/zookeeper-3.5.5/bin/zkServer.sh status;;
        restart)
        /opt/software/zookeeper-3.5.5/bin/zkServer.sh restart;;
        *)echo"require start|stop|status|restart";;
esac

3.2 修改zookeeper文件权限

[root@kafka-01 conf]# chmod 755 /etc/init.d/zookeeper

3.3 添加到服务列表

[root@kafka-01 conf]# chkconfig --add zookeeper

3.4 设置为开机自启动

[root@kafka-01 conf]# chkconfig zookeeper on

4.配置Kafka开机自启动

该自启动方法为早期版本需要节点间提前配置免密，并且该方法存在单点故障，如果自启动的管理节点宕机则无法实现Kafka集群自启动，并且脚本中使用的都是绝对路径，有很大的优化空间

4.1 检查ZK状态脚本

创建检查zookeeper集群健康状态脚本，Kafka集群启动需要确保Zookeeper服务已经启动了才能正常启动，所以在Kafka集群启动前先检查Zookeeper集群状态

[root@kafka-01 ~]# vim AutoStartKafka/zkClusterCheck.sh#!/bin/bash## Check Zookeeper cluster status# Author:fyb# Date:2018/12/14#

zkHosts="kafka-01 kafka-02 kafka-03"function getstatus(){exec 8<>/dev/tcp/$1/2181||continueechostat>&8
MODE=`cat<&8|grep -Po"(?<=Mode: ).*"`exec 8<&-echo${MODE}}for iin$zkHosts;doecho -ne"${i}:"
getstatus${i}done>/var/log/zookeeper/zkStatus.txt 2>/var/log/zookeeper/zkErrors.log

4.2 Kafka集群启动脚本

[root@kafka-01 ~]# vim AutoStartKafka/kafkaStart.sh#!/bin/sh## Get Zookeeper Status,if the zookeeper is started,start Kafka cluster# Author:fyb# Date:2018/12/14#export JAVA_HOME=/opt/software/jdk1.8.0_161export PATH=$JAVA_HOME/bin:$PATH
/root/AutoStartKafka/zkClusterCheck.sh
st=`cat /var/log/zookeeper/zkStatus.txt|grep leader`#定义循环次数变量i
i=0
zkHosts="kafka-01 kafka-02 kafka-03"while[ -z"$st"]do
   /root/AutoStartKafka/zkClusterCheck.sh
   st=`cat /tmp/zkStatus.txt|grep leader`sleep 1let i++if[$i -gt 15];thenfor jin$zkHostsdossh$j"source /etc/profile;/opt/software/zookeeper-3.5.5/bin/zkServer.sh start"donesleep 5fidone
brokerHosts="kafka-01 kafka-02 kafka-03"for kin$brokerHostsdossh$k"source /etc/profile;/opt/software/kafka_2.11-1.1.0/bin/kafka-server-start.sh -daemon /opt/software/kafka_2.11-1.1.0/config/server.properties >/var/log/kafka/KafkaStatus.txt"done

4.3 Kafka集群停止脚本

[root@kafka-01 ~]# vim AutoStartKafka/kafkaStop.sh#!/bin/sh#!/bin/sh## Stop Kafka cluster# Author:fyb# Date:2018/12/14#export JAVA_HOME=/opt/software/jdk1.8.0_161export PATH=$JAVA_HOME/bin:$PATH
brokerHosts="kafka-01 kafka-02 kafka-03"for kin$brokerHostsdossh$k"source /etc/profile;/opt/software/kafka_2.11-1.1.0/bin/kafka-server-stop.sh"done

4.4 添加权限

[root@kafka-01 AutoStartKafka]# chmod 755 zkClusterCheck.sh kafkaStart.sh kafkaStop.sh

4.5 添加kafka服务

[root@kafka-01 ~]# vim /etc/init.d/kafka#!/bin/bashexport JAVA_HOME=/opt/software/jdk1.8.0_161export PATH=$JAVA_HOME/bin:$PATH#chkconfig:2345 20 90#description:kafka#processname:kafkacase$1in
        start)
        /root/AutoStartKafka/kafkaStart.sh;;
        stop)
        /root/AutoStartKafka/kafkaStop.sh;;
        status)
        jps;;
        restart)
        /root/AutoStartKafka/kafkaStop.sh
        /root/AutoStartKafka/kafkaStart.sh;;
        *)echo"require start|stop|status|restart";;
esac

4.6 修改kafka文件权限

[root@kafka-01 ~]# chmod 755 /etc/init.d/kafka

4.7 添加到服务列表

[root@kafka-01 ~]# chkconfig --add kafka

4.8 设置为开机自启动

[root@kafka-01 ~]# chkconfig --add kafka

4.9 检验

重启节点检验功能是否可行，也可以service kafka start检查

  • 作者：脚气水蟑螂药
  • 原文链接：https://blog.csdn.net/fan_yi_bo/article/details/107151475
    更新时间：2022-08-31 13:18:47