Kubernetes 部署 Kafka 集群 「Helm 部署案例」

本项目完全兼容 KRaft, 不依赖 ZooKeeper，最小化的 Kafka 集群仅启动一个 Pod 即可。也可以单独用 Docker 启动，便于本地开发调试，详情请参考 Docker 启动 Kafka

Prerequisites

• Kubernetes 1.18
• Helm 3.3

添加 helm 仓库

代码语言：shell复制

helm repo add kafka-repo https://helm-charts.itboon.top/kafka
helm repo update kafka-repo

helm 部署

部署单节点集群

• 下面这个案例关闭了持久化存储，仅演示部署效果

代码语言：shell复制

helm upgrade --install kafka 
  --namespace kafka-demo 
  --create-namespace 
  --set broker.combinedMode.enabled="true" 
  --set broker.persistence.enabled="false" 
  kafka-repo/kafka

Controller 与 Broker 分离部署

代码语言：shell复制

helm upgrade --install kafka 
  --namespace kafka-demo 
  --create-namespace 
  --set broker.persistence.size="20Gi" 
  kafka-repo/kafka

默认已开启持久化存储。

部署高可用集群

代码语言：shell复制

helm upgrade --install kafka 
  --namespace kafka-demo 
  --create-namespace 
  --set controller.replicaCount="3" 
  --set broker.replicaCount="3" 
  --set broker.heapOpts="-Xms4096m -Xmx4096m" 
  --set broker.resources.requests.memory="8Gi" 
  --set broker.resources.limits.memory="16Gi" 
  kafka-repo/kafka

更多 values 请参考 examples/values-production.yml

LoadBalancer 外部暴露

开启 Kubernetes 集群外访问：

代码语言：shell复制

helm upgrade --install kafka 
  --namespace kafka-demo 
  --create-namespace 
  --set broker.external.enabled="true" 
  --set broker.external.service.type="LoadBalancer" 
  --set broker.external.domainSuffix="kafka.example.com" 
  kafka-repo/kafka

上面部署成功后请完成域名解析配置。

Chart Values

Key	Type	Default	Description
broker.combinedMode.enabled	bool	false	Whether to enable the combined mode

代码语言：yaml复制

broker:
  combinedMode:
    enabled: true
  replicaCount: 1
  heapOpts: "-Xms1024m -Xmx1024m"
  persistence:
    enabled: true
    size: 20Gi

集群外访问

In order to connect to the Kafka server outside the cluster, each Broker must be exposed and advertised.listeners must be correctly configured.

There are two ways to expose, NodePort and LoadBalancer, each broker node needs a NodePort or LoadBalancer.

Chart Values

Key	Type	默认值	描述
broker.external.enabled	bool	false	是否开启集群外访问
broker.external.service.type	string	NodePort	NodePort or LoadBalancer
broker.external.service.annotations	object	{}	External serivce annotations
broker.external.nodePorts	list	[]	NodePort 模式，至少提供一个端口号，如果端口数量少于 broker 数量，则自增
broker.external.domainSuffix	string	kafka.example.com	If you use LoadBalancer for external access, you must use a domain name. The external domain name corresponding to the broker is POD_NAME domain name suffix, such as kafka-broker-0.kafka.example.com. After the deployment, you need to complete the domain name resolution operation

代码语言：yaml复制

## NodePort example
broker:
  replicaCount: 3
  external:
    enabled: true
    service:
      type: "NodePort"
      annotations: {}
    nodePorts:
      - 31050
      - 31051
      - 31052

代码语言：yaml复制

## LoadBalancer example
broker:
  replicaCount: 3
  external:
    enabled: true
    service:
      type: "LoadBalancer"
      annotations: {}
    domainSuffix: "kafka.example.com"

Docker Image

• 项目主页
• GitHub

Docker Compose 启动 Kafka

代码语言：yaml复制

version: "3"

volumes:
  kafka-data: {}

services:
  kafka:
    image: kafkace/kafka:v3.5
    # restart: always
    ports:
      - "29092:29092"
    volumes:
      - kafka-data:/opt/kafka/data
    environment:
      - KAFKA_HEAP_OPTS=-Xmx512m -Xms512m
      - KAFKA_BROKER_EXTERNAL_HOST=kafka.example.com   ## 对外暴露的主机名，可以是域名或IP地址
      - KAFKA_BROKER_EXTERNAL_PORT=29092

  ## kafka web 管理 (可选)
  kafka-ui:
    image: provectuslabs/kafka-ui:v0.7.1
    # restart: always
    ports:
      - "18080:8080"
    environment:
      - KAFKA_CLUSTERS_0_NAME=demo-kafka-server
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092

### 内部网络
## broker 默认内部端口 9092
## bootstrap-server: kafka:9092

### 外部网络
## broker 默认外部端口 29092
## bootstrap-server: ${KAFKA_BROKER_EXTERNAL_HOST}:29092

Environment Variables

变量	默认值	描述
KAFKA_CLUSTER_ID	随机生成	Cluster ID
KAFKA_BROKER_LISTENER_PORT	9092	broker 端口号，如果配置了 KAFKA_CFG_LISTENERS 则此项失效
KAFKA_CONTROLLER_LISTENER_PORT	19091	controller 端口号，如果配置了 KAFKA_CFG_LISTENERS 则此项失效
KAFKA_BROKER_EXTERNAL_HOST	null	对外暴露的主机名，可以是域名或IP地址，如果配置了 KAFKA_CFG_ADVERTISED_LISTENERS 则此项失效
KAFKA_BROKER_EXTERNAL_PORT	29092	对外暴露的端口号，不能跟内部端口重复，如果配置了 KAFKA_CFG_ADVERTISED_LISTENERS 则此项失效
KAFKA_HEAP_OPTS	null	Kafka Java Heap size. 例如: -Xmx512m -Xms512m

https://github.com/itboon/kafka-docker

Kafka Configurations

所有以 KAFKA_CFG_ 开头的环境变量都将映射到其相应的 Apache Kafka 配置项。

例如 KAFKA_CFG_LISTENERS 对应配置参数 listeners，KAFKA_CFG_ADVERTISED_LISTENERS 对应配置参数 advertised.listeners

Variable examples:

变量	配置项
KAFKA_CFG_PROCESS_ROLES	process.roles
KAFKA_CFG_LISTENERS	listeners
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP	listener.security.protocol.map
KAFKA_CFG_ADVERTISED_LISTENERS	advertised.listeners
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS	controller.quorum.voters
KAFKA_CFG_LOG_RETENTION_HOURS	log.retention.hours

log.dir 和 log.dirs 已经被锁定，无法使用环境变量进行覆盖。

其他 Kafka Kubernetes 部署方案

• strimzi-kafka-operator
• Confluent for Kubernetes

kafka apache-kafka kubernetes-helm 容器镜像服务 kuberneteskafka-helm-chart kafka-docke

0 人点赞