Posts 多主机docker部署kafka和zookeeper集群+kowl管理系统
Post
Cancel

多主机docker部署kafka和zookeeper集群+kowl管理系统

Posted 2022年03月16日 2022-03-16T10:48:28+08:00 by Bingal

网上查到的大部分docker部署kafka集群文章都是使用 wurstmeister/zookeeper 和 wurstmeister/kafka 两个镜像，在跨多台机器部署kafka集群的时候遇到些问题，且版本也较旧，没有部署成功。另外，用到的kafka管理工具 sheepkiller/kafka-manager 感觉不是很好用。从docker hub上搜索到被使用较多的是 bitnami/zookeeper 和 bitnami/kafka，所以尝试并成功部署了kafka集群。集群管理工具对比了多款管理工具之后，最好选择了kowl。

3台机器部署kafka集群的分布

192.168.103.1 -> kafka（1个节点）+ zookeeper（一个节点）+ kowl（一个节点） 192.168.103.4 -> kafka（1个节点）+ zookeeper（一个节点） 192.168.103.5 -> kafka（1个节点）+ zookeeper（一个节点）

192.168.103.1 部署方法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# https://hub.docker.com/r/bitnami/kafka
# https://hub.docker.com/r/bitnami/zookeeper
version: '2'
services:
  zoo:
    image: 'bitnami/zookeeper:latest'
    restart: unless-stopped
    hostname: zoo
    container_name: zoo
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    volumes:
      - ./data/zookeeper:/bitnami/zookeeper
    environment:
      ZOO_SERVER_ID: 1
      ZOO_SERVERS: 0.0.0.0:2888:3888,192.168.103.4:2888:3888,192.168.103.5:2888:3888
      ALLOW_ANONYMOUS_LOGIN: yes

  kafka:
    image: 'bitnami/kafka:latest'
    restart: unless-stopped
    hostname: kafka
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.103.1
      KAFKA_HOST_NAME: 192.168.103.1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_BROKER_ID: 1
      KAFKA_CFG_ZOOKEEPER_CONNECT: 192.168.103.1:2181,192.168.103.4:2181,192.168.103.5:2181
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.103.1:9092
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
    volumes:
      - ./data/kafka:/bitnami/kafka

在192.168.103.1机器上，创建docker-compose文件192.168.103.1.yml内容如上，然后执行：

1

docker-compose -f 192.168.103.1.yml up -d

192.168.103.4 部署方法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# https://hub.docker.com/r/bitnami/kafka
# https://hub.docker.com/r/bitnami/zookeeper
version: '2'
services:
  zoo:
    image: 'bitnami/zookeeper:latest'
    restart: unless-stopped
    hostname: zoo
    container_name: zoo
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    volumes:
      - ./data/zookeeper:/bitnami/zookeeper
    environment:
      ZOO_SERVER_ID: 2
      ZOO_SERVERS: 192.168.103.1:2888:3888,0.0.0.0:2888:3888,192.168.103.5:2888:3888
      ALLOW_ANONYMOUS_LOGIN: yes

  kafka:
    image: 'bitnami/kafka:latest'
    restart: unless-stopped
    hostname: kafka
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.103.4
      KAFKA_HOST_NAME: 192.168.103.4
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_BROKER_ID: 2
      KAFKA_CFG_ZOOKEEPER_CONNECT: 192.168.103.1:2181,192.168.103.4:2181,192.168.103.5:2181
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.103.4:9092
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
    volumes:
      - ./data/kafka:/bitnami/kafka

在192.168.103.4机器上，创建docker-compose文件192.168.103.4.yml内容如上，然后执行：

1

docker-compose -f 192.168.103.4.yml up -d

192.168.103.5 部署方法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# https://hub.docker.com/r/bitnami/kafka
# https://hub.docker.com/r/bitnami/zookeeper
version: '2'
services:
  zoo:
    image: 'bitnami/zookeeper:latest'
    restart: unless-stopped
    hostname: zoo
    container_name: zoo
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    volumes:
      - ./data/zookeeper:/bitnami/zookeeper
    environment:
      ZOO_SERVER_ID: 3
      ZOO_SERVERS: 192.168.103.1:2888:3888,192.168.103.4:2888:3888,0.0.0.0:2888:3888
      ALLOW_ANONYMOUS_LOGIN: yes

  kafka:
    image: 'bitnami/kafka:latest'
    restart: unless-stopped
    hostname: kafka
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.103.5
      KAFKA_HOST_NAME: 192.168.103.5
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_BROKER_ID: 3
      KAFKA_CFG_ZOOKEEPER_CONNECT: 192.168.103.1:2181,192.168.103.4:2181,192.168.103.5:2181
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.103.5:9092
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
    volumes:
      - ./data/kafka:/bitnami/kafka

在192.168.103.5机器上，创建docker-compose文件192.168.103.5.yml内容如上，然后执行：

1

docker-compose -f 192.168.103.5.yml up -d

最后在192.168.103.1上部署kowl

创建 docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# 参考 https://github.com/cloudhut/kowl
version: '2'
services:
  kowl:
    image: 'quay.io/cloudhut/kowl:master'
    restart: unless-stopped
    hostname: kowl
    container_name: kowl
    ports:
      - 8085:8085
    volumes:
      - .:/app/conf
    environment:
      CONFIG_FILEPATH: /app/conf/kowl.yaml

因为我用了nginx做前台，需要设置bashpath为/kowl/，并修改了端口为8085（默认8080），所以需要编写kowl的配置文件 kowl-config.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# 参考 https://github.com/cloudhut/kowl/blob/master/docs/config/kowl.yaml
kafka:
  brokers:
    - 192.168.103.1:9092
    - 192.168.103.4:19092
    - 192.168.103.5:19092

server:
  listenPort: 8085
  basePath: "/kowl/"
  readTimeout: 30s
  writeTimeout: 30s
  idleTimeout: 30s
  compressionLevel: 4

把 docker-compose.yml 和 kowl-config.yml 放在同一目录下，然后执行：

1

docker-compose up -d

最后访问 http://192.168.103.1:8085/kowl/ 就可以看到kowl的页面了，默认的账号密码是admin/admin，具体代码已经放到 https://gitee.com/bingal/docker-kafka-zookeeper-kowl

真诚邀请您走进我的知识小宇宙，关注我个人的公众号，在这里，我将不时为您献上独家原创且极具价值的技术内容分享。每一次推送，都倾注了我对技术领域的独特见解与实战心得，旨在与您共享成长过程中的每一份收获和感悟。您的关注和支持，是我持续提供优质内容的最大动力，让我们在学习的道路上并肩同行，共同进步，一起书写精彩的成长篇章!
AI文字转语音
AI个性头像生成
kafka
This post is licensed under CC BY 4.0 by the author.
Recent Update
Contents

jupyter nbconvert 把ipynb文件转换成html格式遇到cdnjs无法访问的问题

画画助手app - 一款iPad上分屏展示参考图的绘画辅助app

Trending Tags