还是在之前安装clickhouse集群的服务器上，使用docker-compose进行kafka集群安装。

安装zookeeper集群

kafka集群运行离不开zookeeper环境，以下是安装zookeeper的docker-compose配置

[root@docker opt]# mkdir zookeeper-cluster
[root@docker opt]# cd zookeeper-cluster/
[root@docker zookeeper-cluster]# vi docker-compose.yml
# 在文件中添加
version: '3.7'
services:
  zoo1:
    image: zookeeper:3.6
    restart: always
    hostname: zoo1
    container_name: zoo1
    ports:
      - 2181:2181
    volumes:
      - ./resource/zoo1/data:/data
      - ./resource/zoo1/datalog:/datalog
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
    networks:
      - net_docker
  zoo2:
    image: zookeeper:3.6
    restart: always
    hostname: zoo2
    container_name: zoo2
    ports:
      - 2182:2181
    volumes:
      - ./resource/zoo2/data:/data
      - ./resource/zoo2/datalog:/datalog
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181
    networks:
      - net_docker
  zoo3:
    image: zookeeper:3.6
    restart: always
    hostname: zoo3
    container_name: zoo3
    ports:
      - 2183:2181
    volumes:
      - ./resource/zoo3/data:/data
      - ./resource/zoo3/datalog:/datalog
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
    networks:
      - net_docker
  zkui:
    image: juris/zkui
    restart: always
    container_name: zkui
    ports:
      - 9099:9090
    environment:
      ZK_SERVER: zoo1:2181,zoo2:2181,zoo3:2181
    networks:
      - net_docker
networks:
  net_docker:
    external: true
[root@docker zookeeper-cluster]# docker-compose up
......

在安装集群时，还安装了一个zookeeper的web可视化工具，在执行docker-comose up后，可通过docker所在宿主机进行访问。

http://宿主机ip:9099 默认用户名/密码： admin/manager

如下图：

kafka集群安装

安装完成zookeeper后，直接进行kafka集群安装：

[root@docker opt]# cd kafka-cluster/
[root@docker kafka-cluster]# vi docker-compose.yml
#在文件中添加
version: '3.7'
networks:
  net_docker:
    external: true
services:
  kafka1:
    image: wurstmeister/kafka:2.12-2.5.0
    restart: always
    hostname: kafka1
    container_name: kafka1
    ports:
      - 9092:9092
    external_links:
      - zoo1
      - zoo2
      - zoo3
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: kafka1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - ./resource/data1:/kafka
    networks:
      - net_docker

  kafka2:
    image: wurstmeister/kafka:2.12-2.5.0
    restart: always
    hostname: kafka2
    container_name: kafka2
    ports:
      - 9093:9093
    external_links:
      - zoo1
      - zoo2
      - zoo3
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_HOST_NAME: kafka2
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9093
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - ./resource/data2:/kafka
    networks:
      - net_docker

  kafka3:
    image: wurstmeister/kafka:2.12-2.5.0
    restart: always
    hostname: kafka3
    container_name: kafka3
    ports:
      - 9094:9094
    external_links:
      - zoo1
      - zoo2
      - zoo3
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ADVERTISED_HOST_NAME: kafka3
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094
      KAFKA_LISTENERS: PLAINTEXT://kafka3:9094
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
    volumes:
      - ./resource/data3:/kafka
    networks:
      - net_docker

  kafka-manager:
    image: sheepkiller/kafka-manager:latest
    restart: always
    container_name: kafka-manager
    hostname: kafka-manager
    ports:
      - 9009:9000
    links:
      - kafka1
      - kafka2
      - kafka3
    external_links:
      - zoo1
      - zoo2
      - zoo3
    environment:
      ZK_HOSTS: zoo1:2181,zoo2:2181,zoo3:2181
      TZ: CST-8
    networks:
      - net_docker
[root@docker kafka-cluster]# docker-compose up
......

在安装kafka过程中，也安装了kafka-manager工具，可通过宿主机+9009端口进行访问。

http://宿主机ip:9009

如下图：

配置kafka集群

选择下拉框中，Add_Cluster 后进行填写：

• Cluster Name 自定义名称
• Cluster Zookeeper Host zookeeper的主机名。

配置完成后如下图所示：

配置完成后，点击页面下方的 Save按钮。

kafka、clickhouse集群集成

1. 如果kafka、zookeeper 没有和clickhouse在同一个docker网络下，那么，需要在clickhouse的docker-compose.yml文件中，配置如下参数：

  extra_hosts:
      - kafka1:kafka宿主机ip
      - kafka2:kafka宿主机ip
      - kafka3:kafka宿主机ip
      - zoo1:zookeeper宿主机ip
      - zoo2:zookeeper宿主机ip
      - zoo3:zookeeper宿主机ip

目前，我所搭建的测试环境不用配置。

1. 修改clickhouse的metrika.xml文件内容，在文件中添加zookeeper相关配置数据。

在<yandex>标签内添加如下内容2个clickhouse节点都需要配置：

#节点1
[root@docker cluster-ch]# vi docker_compose_data/clickhouse-server1/metrika.xml
<zookeeper-servers>
        <node index="1">
            <host>zoo1</host>
            <port>2181</port>
        </node>
        <node index="2">
            <host>zoo2</host>
            <port>2182</port>
        </node>
        <node index="3">
            <host>zoo3</host>
            <port>2183</port>
        </node>
    </zookeeper-servers>
#节点2
[root@docker cluster-ch]# vi docker_compose_data/clickhouse-server2/metrika.xml
......

修改后，重启clickhouse集群：

[root@docker cluster-ch]# docker-compose stop
Stopping clickHouse1 ... done
Stopping clickHouse2 ... done
[root@docker cluster-ch]# docker-compose up
......

至此，安装完成。

下一篇，将记录fluentd，接收日志并格式化输出到kafka集群，最后加载到clickhouse集群中。