{"id":5311,"date":"2022-04-19T22:26:17","date_gmt":"2022-04-19T13:26:17","guid":{"rendered":"https:\/\/www.skyer9.pe.kr\/wordpress\/?p=5311"},"modified":"2022-04-19T19:10:50","modified_gmt":"2022-04-19T10:10:50","slug":"kafka-%ec%84%a4%ec%b9%98on-aws-%eb%b0%8f-%eb%b2%a4%ec%b9%98%eb%a7%88%ed%82%b9-2","status":"publish","type":"post","link":"https:\/\/www.skyer9.pe.kr\/wordpress\/?p=5311","title":{"rendered":"Kafka \uc124\uce58(on AWS) \ubc0f \ubca4\uce58\ub9c8\ud0b9"},"content":{"rendered":"<h1>Kafka \uc124\uce58(on AWS) \ubc0f \ubca4\uce58\ub9c8\ud0b9<\/h1>\n<p>\uc6b0\uc120 \ud558\ub098\uc758 \uba38\uc2e0\uc744 \uc0dd\uc131\ud558\uace0, \uacf5\ud1b5\uc124\uc815\uc744 \ub05d\ub0b8 \ud6c4, \uc774\ubbf8\uc9c0\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4.<br \/>\n\uc800\uc7a5\ub41c \uc774\ubbf8\uc9c0\ub85c 2\uac1c\uc758 \uc778\uc2a4\ud134\uc2a4\ub97c \ucd94\uac00\ub85c \uc0dd\uc131\ud558\uace0 \ud074\ub7ec\uc2a4\ud130\ub97c \uc2e4\ud589\ud569\ub2c8\ub2e4.<\/p>\n<p>\ub178\ub4dc \uc0dd\uc131 \ud6c4 Kafka \uc5d0 \ud3ec\ud568\ub418\uc5b4 \uc788\ub294 Benchmark \ud234\uc744 \uc774\uc6a9\ud574 \uc131\ub2a5 \ud14c\uc2a4\ud2b8\ub97c \ud569\ub2c8\ub2e4.<\/p>\n<h2>AWS EC2 \uc0dd\uc131<\/h2>\n<p>OS \ub294 Ubuntu 64bit \ub85c \ud569\ub2c8\ub2e4.<br \/>\n\uc778\uc2a4\ud134\uc2a4 \uc720\ud615 \uc740 m5a.large \ub85c \uc120\ud0dd\ud569\ub2c8\ub2e4.(\uba54\ubaa8\ub9ac 8\uae30\uac00)<\/p>\n<h2>JDK \uc124\uce58<\/h2>\n<pre><code class=\"language-bash\">sudo apt-get update\nsudo apt-get install openjdk-11-jdk<\/code><\/pre>\n<h2>Kafka \uacc4\uc815\uc0dd\uc131<\/h2>\n<pre><code class=\"language-bash\">sudo adduser kafka\nsudo su - kafka<\/code><\/pre>\n<h2>Kafka \uc124\uce58<\/h2>\n<p><a href=\"https:\/\/dlcdn.apache.org\/kafka\/\">https:\/\/dlcdn.apache.org\/kafka\/<\/a> \uc5d0\uc11c \uc6d0\ud558\ub294 Kafka \ubc84\uc804\uc744 \ub2e4\uc6b4\ubc1b\uc2b5\ub2c8\ub2e4.<br \/>\n(\uc5ec\uae30\uc11c\ub294 2.8.1 \uc744 \uc120\ud0dd\ud569\ub2c8\ub2e4.)<\/p>\n<pre><code class=\"language-bash\">wget https:\/\/dlcdn.apache.org\/kafka\/2.8.1\/kafka_2.13-2.8.1.tgz\ntar xvfz kafka_2.13-2.8.1.tgz<\/code><\/pre>\n<pre><code class=\"language-bash\">vi .bashrc\n......\nexport KAFKA_HOME=\/home\/kafka\/kafka_2.13-2.8.1\n......\n\nexit\nsudo su - kafka<\/code><\/pre>\n<pre><code class=\"language-bash\">mkdir $KAFKA_HOME\/logs\nmkdir $KAFKA_HOME\/data<\/code><\/pre>\n<h2>hosts \ud30c\uc77c \uc218\uc815<\/h2>\n<p>\uc6b0\uc120\uc740 \ud558\ub098\uc758 \ub178\ub4dc\ub9cc \uc0dd\uc131\uc911\uc774\ubbc0\ub85c \uc0dd\uc131\ud55c \uc778\uc2a4\ud134\uc2a4\uc758 Private IP \ub97c \ubaa8\ub450 \uc801\uc5b4\uc900\ub2e4.<\/p>\n<pre><code class=\"language-bash\">sudo vi \/etc\/hosts\n......\n172.31.28.2   node01\n172.31.28.2   node02\n172.31.28.2   node03\n......<\/code><\/pre>\n<h2>Kafka \ucd94\uac00\uc124\uc815<\/h2>\n<pre><code class=\"language-bash\">vi $KAFKA_HOME\/config\/server.properties\n......\nbroker.id=1\n......\nlisteners=PLAINTEXT:\/\/node01:9092\n......\nlog.dirs=\/home\/kafka\/kafka_2.13-2.8.1\/logs\n......\nzookeeper.connect=node01:2181,node02:2181,node03:2181<\/code><\/pre>\n<pre><code class=\"language-bash\">echo &quot;1&quot; &gt; $KAFKA_HOME\/data\/myid\n\nvi $KAFKA_HOME\/config\/zookeeper.properties\n......\ndataDir=\/home\/kafka\/kafka_2.13-2.8.1\/data\n......\ninitLimit=5\nsyncLimit=2\nserver.1=node01:2888:3888\nserver.2=node02:2888:3888\nserver.3=node03:2888:3888\n......<\/code><\/pre>\n<h2>ssh \uc124\uc815\ucd94\uac00(\ube44\ubc00\ubc88\ud638 \uc5c6\uc774 \uac01 \uc11c\ubc84 \uc811\uc18d)<\/h2>\n<pre><code class=\"language-bash\">ssh-keygen -t rsa -P &#039;&#039; -f ~\/.ssh\/id_rsa\ncat ~\/.ssh\/id_rsa.pub &gt;&gt; ~\/.ssh\/authorized_keys\nchmod 0600 ~\/.ssh\/authorized_keys<\/code><\/pre>\n<h2>\ubcf4\uc548\uadf8\ub8f9 \uc0dd\uc131 \ubc0f \uc801\uc6a9<\/h2>\n<p>kafka \ubcf4\uc548\uadf8\ub8f9\uc744 \uc0dd\uc131\ud569\ub2c8\ub2e4.<br \/>\n\uc0dd\uc131\ud55c \ubcf4\uc548\uadf8\ub8f9\uc5d0 kafka \ub85c\ubd80\ud130\uc758 \ubaa8\ub4e0 \ud2b8\ub798\ud53d\uc744 \ud5c8\uc6a9\ud574 \uc90d\ub2c8\ub2e4.<\/p>\n<h2>\uc774\ubbf8\uc9c0 \uc0dd\uc131<\/h2>\n<p>\ud604\uc7ac\uae4c\uc9c0 \uc791\uc5c5\ud55c \uc0c1\ud0dc\ub85c \uc774\ubbf8\uc9c0\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4.<br \/>\n\uc774\ubbf8\uc9c0 \uc0dd\uc131\uc5d0\ub294 5\ubd84\uc815\ub3c4 \uc18c\uc694\ub429\ub2c8\ub2e4.<\/p>\n<h2>\uc778\uc2a4\ud134\uc2a4 \uc0dd\uc131<\/h2>\n<p>\uc0dd\uc131\ud55c \uc774\ubbf8\uc9c0\ub85c 2\uac1c\uc758 \uc778\uc2a4\ud134\uc2a4\ub97c \ucd94\uac00\ub85c \uc0dd\uc131\ud569\ub2c8\ub2e4.<br \/>\n3\uac1c\uc758 \uc778\uc2a4\ud134\uc2a4 \ud638\uc2a4\ud2b8 \ud30c\uc77c\uc744 \ubaa8\ub450 \uc218\uc815\ud574 \uc90d\ub2c8\ub2e4.<\/p>\n<pre><code class=\"language-bash\">sudo vi \/etc\/hosts\n......\n172.31.28.2     node01\n172.31.24.232   node02\n172.31.19.100   node03\n......<\/code><\/pre>\n<h2>\uc778\uc2a4\ud134\uc2a4 \uc124\uc815\uc218\uc815<\/h2>\n<pre><code class=\"language-bash\">ssh node02\necho &quot;2&quot; &gt; $KAFKA_HOME\/data\/myid\nvi $KAFKA_HOME\/config\/server.properties\n......\nbroker.id=2\n......\n\nssh node03\necho &quot;3&quot; &gt; $KAFKA_HOME\/data\/myid\nvi $KAFKA_HOME\/config\/server.properties\n......\nbroker.id=3\n......<\/code><\/pre>\n<h2>\uc778\uc2a4\ud134\uc2a4 \uc2e4\ud589\ud558\uae30<\/h2>\n<pre><code class=\"language-bash\">sudo su - kafka\n\nssh node01\n$KAFKA_HOME\/bin\/zookeeper-server-start.sh $KAFKA_HOME\/config\/zookeeper.properties &amp;\n$KAFKA_HOME\/bin\/kafka-server-start.sh $KAFKA_HOME\/config\/server.properties &amp;\n\nssh node02\n$KAFKA_HOME\/bin\/zookeeper-server-start.sh $KAFKA_HOME\/config\/zookeeper.properties &amp;\n$KAFKA_HOME\/bin\/kafka-server-start.sh $KAFKA_HOME\/config\/server.properties &amp;\n\nssh node03\n$KAFKA_HOME\/bin\/zookeeper-server-start.sh $KAFKA_HOME\/config\/zookeeper.properties &amp;\n$KAFKA_HOME\/bin\/kafka-server-start.sh $KAFKA_HOME\/config\/server.properties &amp;<\/code><\/pre>\n<h2>\uc2a4\ud2b8\ub798\uc2a4 \ud14c\uc2a4\ud2b8<\/h2>\n<p>\ub808\ucf54\ub4dc 1000\uac1c<\/p>\n<pre><code class=\"language-bash\">export KAFKA_HEAP_OPTS=&quot;-Xmx2048M&quot;\n$KAFKA_HOME\/bin\/kafka-producer-perf-test.sh \\\n    --topic test \\\n    --num-records 1000 \\\n    --throughput -1 \\\n    --producer-props \\\n    bootstrap.servers=localhost:9092 \\\n    batch.size=1000 acks=1 linger.ms=100000 \\\n    buffer.memory=4294967296 \\\n    request.timeout.ms=300000 \\\n    --record-size 1000<\/code><\/pre>\n<p>\ub808\ucf54\ub4dc 10000\uac1c<\/p>\n<pre><code class=\"language-bash\">export KAFKA_HEAP_OPTS=&quot;-Xmx2048M&quot;\n$KAFKA_HOME\/bin\/kafka-producer-perf-test.sh \\\n    --topic test \\\n    --num-records 10000 \\\n    --throughput -1 \\\n    --producer-props \\\n    bootstrap.servers=localhost:9092 \\\n    batch.size=1000 acks=1 linger.ms=100000 \\\n    buffer.memory=4294967296 \\\n    request.timeout.ms=300000 \\\n    --record-size 1000<\/code><\/pre>\n<p>\ub808\ucf54\ub4dc 1000000\uac1c<\/p>\n<pre><code class=\"language-bash\">export KAFKA_HEAP_OPTS=&quot;-Xmx2048M&quot;\n$KAFKA_HOME\/bin\/kafka-producer-perf-test.sh \\\n    --topic test \\\n    --num-records 1000000 \\\n    --throughput -1 \\\n    --producer-props \\\n    bootstrap.servers=localhost:9092 \\\n    batch.size=1000 acks=1 linger.ms=100000 \\\n    buffer.memory=4294967296 \\\n    request.timeout.ms=300000 \\\n    --record-size 1000<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Kafka \uc124\uce58(on AWS) \ubc0f \ubca4\uce58\ub9c8\ud0b9 \uc6b0\uc120 \ud558\ub098\uc758 \uba38\uc2e0\uc744 \uc0dd\uc131\ud558\uace0, \uacf5\ud1b5\uc124\uc815\uc744 \ub05d\ub0b8 \ud6c4, \uc774\ubbf8\uc9c0\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4. \uc800\uc7a5\ub41c \uc774\ubbf8\uc9c0\ub85c 2\uac1c\uc758 \uc778\uc2a4\ud134\uc2a4\ub97c \ucd94\uac00\ub85c \uc0dd\uc131\ud558\uace0 \ud074\ub7ec\uc2a4\ud130\ub97c \uc2e4\ud589\ud569\ub2c8\ub2e4. \ub178\ub4dc \uc0dd\uc131 \ud6c4 Kafka \uc5d0 \ud3ec\ud568\ub418\uc5b4 \uc788\ub294 Benchmark \ud234\uc744 \uc774\uc6a9\ud574 \uc131\ub2a5 \ud14c\uc2a4\ud2b8\ub97c \ud569\ub2c8\ub2e4. AWS EC2 \uc0dd\uc131 OS \ub294 Ubuntu 64bit \ub85c \ud569\ub2c8\ub2e4. \uc778\uc2a4\ud134\uc2a4 \uc720\ud615 \uc740 m5a.large \ub85c \uc120\ud0dd\ud569\ub2c8\ub2e4.(\uba54\ubaa8\ub9ac 8\uae30\uac00) JDK \uc124\uce58 sudo apt-get\u2026 <span class=\"read-more\"><a href=\"https:\/\/www.skyer9.pe.kr\/wordpress\/?p=5311\">Read More &raquo;<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-5311","post","type-post","status-publish","format-standard","hentry","category-kafka"],"_links":{"self":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/5311","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5311"}],"version-history":[{"count":9,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/5311\/revisions"}],"predecessor-version":[{"id":5320,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/5311\/revisions\/5320"}],"wp:attachment":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5311"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5311"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5311"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}