Kafka 모니터링

Prometheus and Grafana 를 이용한 Kafka 모니터링 방법을 설명합니다.

Prometheus JMX exporter 설치하기

Kafka 가 설치되어 있는 서버에서 아래 설정을 합니다.

sudo mkdir /opt/kafka/prometheus/
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-2_0_0.yml

sudo vi /opt/kafka/prometheus/kafka-2_0_0.yml

- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w*
  name: kafka_producer_$2
- pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w*
  name: kafka_consumer_$2
- pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w*
  name: kafka_consumer_$2

sudo vi /opt/kafka/bin/kafka-server-start.sh

export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7071:/opt/kafka/prometheus/kafka-2_0_0.yml"

sudo vi /opt/kafka/bin/kafka-console-consumer.sh

export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7072:/opt/kafka/prometheus/kafka-2_0_0.yml"

sudo vi /opt/kafka/bin/kafka-console-producer.sh

export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7073:/opt/kafka/prometheus/kafka-2_0_0.yml"

sudo systemctl restart kafka.service

Prometheus 설치하기

sudo mkdir /etc/prometheus /var/lib/prometheus /var/log/prometheus /var/run/prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.2.1/prometheus-2.2.1.linux-amd64.tar.gz -P /tmp/
sudo tar -zxvf /tmp/prometheus-2.2.1.linux-amd64.tar.gz -C /tmp/
sudo cp /tmp/prometheus-2.2.1.linux-amd64/{prometheus,promtool} /usr/local/bin/
sudo chmod +x /usr/local/bin/{prometheus,promtool}
sudo cp /tmp/prometheus-2.2.1.linux-amd64/prometheus.yml /etc/prometheus/
sudo cp -r /tmp/prometheus-2.2.1.linux-amd64/{consoles,console_libraries} /etc/prometheus/

sudo vi /etc/prometheus/prometheus.yml

  - job_name: 'kafka-server'
    static_configs:
      - targets: ['172.31.31.223:7071']
  # - job_name: 'kafka-consumer'
  #   static_configs:
  #     - targets: ['172.31.31.223:7072']
  # - job_name: 'kafka-producer'
  #   static_configs:
  #     - targets: ['172.31.31.223:7073']

sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/prometheus \
         --config.file /etc/prometheus/prometheus.yml \
         --storage.tsdb.path /var/lib/prometheus/ \
         --web.console.templates=/etc/prometheus/consoles \
         --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target

sudo systemctl start prometheus.service
sudo systemctl enable prometheus.service

http://<서버아이피>:9090/ 에 접속하면 UI 를 볼 수 있습니다.