Posts 多主机docker部署kafka和zookeeper集群+kowl管理系统
Post
Cancel

多主机docker部署kafka和zookeeper集群+kowl管理系统

网上查到的大部分docker部署kafka集群文章都是使用 wurstmeister/zookeeper 和 wurstmeister/kafka 两个镜像,在跨多台机器部署kafka集群的时候遇到些问题,且版本也较旧,没有部署成功。另外,用到的kafka管理工具 sheepkiller/kafka-manager 感觉不是很好用。从docker hub上搜索到被使用较多的是 bitnami/zookeeper 和 bitnami/kafka,所以尝试并成功部署了kafka集群。集群管理工具对比了多款管理工具之后,最好选择了kowl。

3台机器部署kafka集群的分布

192.168.103.1 -> kafka(1个节点)+ zookeeper(一个节点)+ kowl(一个节点) 192.168.103.4 -> kafka(1个节点)+ zookeeper(一个节点) 192.168.103.5 -> kafka(1个节点)+ zookeeper(一个节点)

192.168.103.1 部署方法:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# https://hub.docker.com/r/bitnami/kafka
# https://hub.docker.com/r/bitnami/zookeeper
version: '2'
services:
  zoo:
    image: 'bitnami/zookeeper:latest'
    restart: unless-stopped
    hostname: zoo
    container_name: zoo
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    volumes:
      - ./data/zookeeper:/bitnami/zookeeper
    environment:
      ZOO_SERVER_ID: 1
      ZOO_SERVERS: 0.0.0.0:2888:3888,192.168.103.4:2888:3888,192.168.103.5:2888:3888
      ALLOW_ANONYMOUS_LOGIN: yes

  kafka:
    image: 'bitnami/kafka:latest'
    restart: unless-stopped
    hostname: kafka
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.103.1
      KAFKA_HOST_NAME: 192.168.103.1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_BROKER_ID: 1
      KAFKA_CFG_ZOOKEEPER_CONNECT: 192.168.103.1:2181,192.168.103.4:2181,192.168.103.5:2181
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.103.1:9092
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
    volumes:
      - ./data/kafka:/bitnami/kafka

在192.168.103.1机器上,创建docker-compose文件192.168.103.1.yml内容如上,然后执行:

1
docker-compose -f 192.168.103.1.yml up -d

192.168.103.4 部署方法:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# https://hub.docker.com/r/bitnami/kafka
# https://hub.docker.com/r/bitnami/zookeeper
version: '2'
services:
  zoo:
    image: 'bitnami/zookeeper:latest'
    restart: unless-stopped
    hostname: zoo
    container_name: zoo
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    volumes:
      - ./data/zookeeper:/bitnami/zookeeper
    environment:
      ZOO_SERVER_ID: 2
      ZOO_SERVERS: 192.168.103.1:2888:3888,0.0.0.0:2888:3888,192.168.103.5:2888:3888
      ALLOW_ANONYMOUS_LOGIN: yes

  kafka:
    image: 'bitnami/kafka:latest'
    restart: unless-stopped
    hostname: kafka
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.103.4
      KAFKA_HOST_NAME: 192.168.103.4
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_BROKER_ID: 2
      KAFKA_CFG_ZOOKEEPER_CONNECT: 192.168.103.1:2181,192.168.103.4:2181,192.168.103.5:2181
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.103.4:9092
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
    volumes:
      - ./data/kafka:/bitnami/kafka

在192.168.103.4机器上,创建docker-compose文件192.168.103.4.yml内容如上,然后执行:

1
docker-compose -f 192.168.103.4.yml up -d

192.168.103.5 部署方法:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# https://hub.docker.com/r/bitnami/kafka
# https://hub.docker.com/r/bitnami/zookeeper
version: '2'
services:
  zoo:
    image: 'bitnami/zookeeper:latest'
    restart: unless-stopped
    hostname: zoo
    container_name: zoo
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    volumes:
      - ./data/zookeeper:/bitnami/zookeeper
    environment:
      ZOO_SERVER_ID: 3
      ZOO_SERVERS: 192.168.103.1:2888:3888,192.168.103.4:2888:3888,0.0.0.0:2888:3888
      ALLOW_ANONYMOUS_LOGIN: yes

  kafka:
    image: 'bitnami/kafka:latest'
    restart: unless-stopped
    hostname: kafka
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.103.5
      KAFKA_HOST_NAME: 192.168.103.5
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_BROKER_ID: 3
      KAFKA_CFG_ZOOKEEPER_CONNECT: 192.168.103.1:2181,192.168.103.4:2181,192.168.103.5:2181
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.103.5:9092
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
    volumes:
      - ./data/kafka:/bitnami/kafka

在192.168.103.5机器上,创建docker-compose文件192.168.103.5.yml内容如上,然后执行:

1
docker-compose -f 192.168.103.5.yml up -d

最后在192.168.103.1上部署kowl

创建 docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 参考 https://github.com/cloudhut/kowl
version: '2'
services:
  kowl:
    image: 'quay.io/cloudhut/kowl:master'
    restart: unless-stopped
    hostname: kowl
    container_name: kowl
    ports:
      - 8085:8085
    volumes:
      - .:/app/conf
    environment:
      CONFIG_FILEPATH: /app/conf/kowl.yaml

因为我用了nginx做前台,需要设置bashpath为/kowl/,并修改了端口为8085(默认8080),所以需要编写kowl的配置文件 kowl-config.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 参考 https://github.com/cloudhut/kowl/blob/master/docs/config/kowl.yaml
kafka:
  brokers:
    - 192.168.103.1:9092
    - 192.168.103.4:19092
    - 192.168.103.5:19092

server:
  listenPort: 8085
  basePath: "/kowl/"
  readTimeout: 30s
  writeTimeout: 30s
  idleTimeout: 30s
  compressionLevel: 4

把 docker-compose.yml 和 kowl-config.yml 放在同一目录下,然后执行:

1
docker-compose up -d

最后访问 http://192.168.103.1:8085/kowl/ 就可以看到kowl的页面了,默认的账号密码是admin/admin,具体代码已经放到 https://gitee.com/bingal/docker-kafka-zookeeper-kowl


邀请您关注我的公众号,我将会不时地为您推送独家原创的技术内容分享。
This post is licensed under CC BY 4.0 by the author.

jupyter nbconvert 把ipynb文件转换成html格式遇到cdnjs无法访问的问题

画画助手app - 一款iPad上分屏展示参考图的绘画辅助app

Trending Tags